Tensordyne co-founder and CPO R K Anand joins Austin to discuss the company's strategy for disrupting AI inference. RK explains how Tensordyne combines power-efficient logarithmic math with a battle-hardened networking fabric from partner HPE Juniper. The result is a high-density, air-cooled system designed to efficiently run massive Mixture-of-Experts models in existing data centers. Key Takeaways: - The core innovation isn't just log math, it's the patented method for accumulation. This turns expensive multiplications into cheap additions, freeing die space for a massive on-chip SRAM cache. - Networking is a partnership, not a project. Tensordyne leverages HPE Juniper's 7th-gen router fabric, skipping development cycles to get a 1-2 microsecond latency solution ideal for random MoE traffic. - The power and density claims are radical. By combining log math silicon with an air-cooled fabric, Tensordyne packs 72 chips into a 13U chassis at just 30 kW - a quarter of the space and power of an NVL72. - One go-to-market advantage is air cooling. The 30 kW, 19-inch rack system can be deployed in existing 'brownfield' enterprise and telco data centers that cannot support liquid cooling. - Partnerships de-risk the aggressive timeline. Broadcom provides access to TSMC 3nm and HBM, while strategic investor HPE Juniper provides the carrier-grade fabric with 'five nines' reliability. Chapters: 0:00 Introducing Tensordyne 5:32 The Juniper vs. Cisco Playbook 11:29 Origin Story: Automotive Power Constraints 15:37 The Secret Sauce of Log Math 18:02 Pivoting to the Data Center 22:08 Leveraging a Router Backplane for AI 27:22 Why Router Fabrics Suit MoE Models 34:12 The Three Phases of Inference Hardware 37:40 How One Chip Handles Pre-fill & Decode 40:34 The 'Too Good to Be True' System Specs 43:31 Go-to-Market: The Air-Cooled Advantage 48:21 De-risking with Strategic Partnerships 52:37 Solving the Software Problem with AI Follow Chipstrat: Newsletter: https://www.chipstrat.com X: https://x.com/chipstrat Follow Vik: Newsletter: https://www.viksnewsletter.com/ X: https://x.com/vikramskr Follow Semi Doped: Get more of Austin and Vik daily, free: https://daily.semidoped.com/ - The software moat is eroding. Tensordyne argues that modern agentic AI workflows can now automate the generation of optimized software kernels, solving the classic adoption problem for new hardware.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_00Hey everyone, welcome back to another semi-dope podcast. Today we've got a special guest, RK Anon, co-founder and chief product officer at Tensordyne. If you haven't heard of Tensordyne, Tensordyne is building an inference chip and inference rack that does its math in a completely different number system and has other interesting architectural tricks that we'll talk about. So, RK, I'm pleased to have you here to talk more about Tensordyne.
SPEAKER_01Austin, it's a pleasure to meet you. And uh I'm very glad to uh be here. Uh I watch a lot of your podcasts and I'm a big fan. So I'm I'm glad to be here today.
SPEAKER_00Oh, right on. Awesome. Awesome. I love that. Um thank you. Okay, so cool. So let's start with you. Um, you have a fascinating background in networking, which is really interesting because I think now that we're at Rack Scale, obviously networking is a big part of the AI inference story. So, you know, maybe tell our listeners more about you. Take us way back. You were at Sun, you were a founding engineer at Juniper. Kind of tell us about your career arc that led you all the way here.
SPEAKER_01Um yeah, so you know, uh, this is my 37th year uh in Silicon Valley. I started my career at Sun uh in 1990, actually, uh designing microprocessors. So I was part of a microprocessor team. And Sun was building high-end servers uh for the market. So it was a really fascinating time. There's a lot of competition, there was RISC versus Cisq, uh, and you got exposed to building really large um multiprocessing systems. Uh fast forward a few years, so 1996, I was uh fortunate to join uh Juniper Networks uh as one of the founding engineers and led the first uh silicon efforts there.
SPEAKER_01And uh the reason I was able to do that was because Pradeep Sindhu, who's the founder of Juniper, uh uh was one of my early mentors at Sun. It was part of a team from Xerox Palo Alto Research Center helping us build these big machines. And he um gave me the opportunity to join Juniper. Um and I spent almost 17 years there with a little bit of a break, um, and saw the company from zero slightly over four and a quarter billion dollars of revenue. And the most uh fascinating part was that we were uh in the early days of the internet.
SPEAKER_01Uh the internet was doubling every six to nine months, and I think Juniper I think transformed networking pretty significantly. So it's a very good learning opportunity, uh, opportunity to start from scratch, uh, grow with the company, and uh and eventually be very successful in building the biggest routers in the world. But most importantly, I think building great relationships and great friendships and being able to mentor a lot of the talent there. So I think it's a it's a great learning experience for me.
SPEAKER_00Oh man, it sounds like it. Okay, so interesting. So you got to work on compute back at Sun, and then you got to work on networking for a long time at Juniper, and then here we are in this era, which is both compute and networking. Uh so I it seems like you have learned all the right things that you need for this era.
SPEAKER_01Yeah, it's uh it's fascinating. You know, I think the Silicon Valley, if you've been here long enough, goes through cycles, right? And we're in one more of those really great innovation cycles. And yeah, you're right. Uh, you know, computing, you think about computing and you say, okay, general purpose computing or high-end client server computing. And how was that built? Uh, you know, because we started from PCs and, you know, mainframes, then PCs, and they went to client-server. And Sun was a premier company, a great innovation, a great place for innovating and building uh great technology.
SPEAKER_01And then Juniper ends up being the premier company in networking, building high-end core and edge routers, right? The fact that we have the internet today uh that works at so flawlessly is because of all the work that companies like Juniper, Cisco, and others did. Um, and then we come back, come to the world of AI, and it's uh again back to computing, but uh then we start hearing for the last couple, three years, like networking is so central to AI. So it's a coming together of all of these things, and you know, you can then reflect back and see if you can take those learnings and apply them to what you're doing today, I would say.
SPEAKER_00Sure, yeah, totally. So, what I think is interesting, and uh, you know, correct me anywhere I'm wrong, but when you guys started at Juniper, I mean I assume that it was Cisco was probably the dominant player in the in the space.
SPEAKER_01They were. Uh it's uh you know, Cisco was started in the early 80s and by uh probably mid 80s, I don't remember the exact time, but when Juniper started in 1996, uh the browser had just appeared, right? Uh out of Urbano Champagne and Netscape, the Mosaic browser and the Netscape had appeared, and we were trying to put a lot of our data on the internet, and we were trying to start sharing our think of it almost like sharing your hard drives, basically. And and they were uh they were these internet service providers, like UUNet was an example of them, and and they were uh they were using Cisco routers, and Cisco was the dominant player.
SPEAKER_01I think Cisco had like 93, 94 percent of the core router market share. And uh fortunately for us, Padeep, who was the founder of Juniper, looked at the way the routers were getting built, and he was trying to wonder why they are not getting built like high-end servers. So he had a general thesis that he could actually build routers far more efficiently. And and then you uh then we got get to hear from people like Mike Odell, who was CTO at Uinet, that the internet was doubling every six to nine months, and he was not able to keep up. And uh and so Juniper fundamentally uh re-architected how routers were built, and that uh completely transformed how the growth of um the internet happened.
SPEAKER_01Um Cisco, by the year, sometime 2000, was the number one market cap company in the world, you know, they were like, and uh and John Chambers was on the cover of every magazine in the world. Uh and and Juniper appeared in the landscape, and we built the highest performing router. And then I think within the span of a year, a couple of years, we had 30% market share. So there is proof positive that when new markets evolve, there are opportunities for startups to come in. And if they have technology that's differentiated, uh, they can have an impact on the market.
SPEAKER_00Nice. Awesome. Yes, you you took it exactly where I was thinking in my head, which is like, wow, this actually feels very similar to back then where there's a new um technological revolution happening that's impacting the end users, which was the internet at the time and it's LLMs today. Um, and there there's an incumbent that, of course, uh, you know, is the largest market cap company in the world and is quite dominant. And I'm sure when you guys started Juniper, there was probably a lot of people that just said, like, what are you doing?
SPEAKER_00How could you ever possibly compete with Cisco? Um, but then what I heard you say was the interesting insight is like, well, if you think that there's a different way to tackle the technology, that ultimately you do think you can compete on technical performance and then probably TCO and other things.
SPEAKER_01Yeah, uh and you know, of course the paroles only uh quickly end there because the size and scale of the market is kind of way, way different now. Um and and the competitive landscape, uh, you know, I mean, though when in those days of the internet, you know, information was more constrained. Everybody knows we live in a different world today. Information is instantaneous, everybody knows everybody else. Uh, but in that period, between say 96 and 99, there were many other companies that were funded to build uh routers.
SPEAKER_01Uh, I mean, all those names are etched in my mind, right? There's Avici and there's Fluuris and Nexabit, and there's a bunch of companies. So uh there's an opportunity that a landscape opens up, an opportunity opens up. I think if you look at today, the scale is orders of magnitude larger. The you know, the market capitalization of some of these companies is quite significant, and then the uh and the technology power that is there, especially with um Google or a lot of the hyperscalers also building silicon and systems, is that there's a lot more information.
SPEAKER_01Um but I think there's a there's a there's a there's a you know, just like anything else, there are waves and their changes that happen. I think that if you if you look at the last five, six years, uh the initial phase was all training, AI training, right? And then we have slightly shifted from training to inference. And inference was initially we were doing monolithic kind of models, and now we're in the world of two trillion plus MOEs. So the the architecture demands for inference are different from training.
SPEAKER_01And then for monolithic models to MOEs with you know two plus trillion models or 2.8 like Kimi K3, the pressures on the system are different. So there's a window of opportunity that opens up on if you think about it, and how can you build a system that addresses demand for today's market that looks different from something that's four or five years back?
SPEAKER_00Yes, perfect. Okay, so lead us into from there then into Tensordyne. And what we we have this window here, it's it's inference at just kind of massive scale. Um it's mixture of experts, two trillion parameters. We want high interactivity these days. It's agentic AI, right? So agents want to run at thousands of tokens per second, ideally. Um and so therefore, so we've got this gap, we've got this window. There is definitely a way that everyone's doing it now, which is mostly like running this inference on racks and racks of GPUs.
SPEAKER_00And Tensor9 is taking a different approach. So, like, tell our listeners what is the technical differentiation that you guys are pursuing that you think is a different bet.
SPEAKER_01If if one were to break down the problem, uh you know, the models are immensely large. They, you know, so they don't fit on single devices anymore. So so now you have to have many devices that somehow operate in a kind of a synchronous or orchestrated mode. Well, how do they do that? Well, you have to connect them together. So there's a networking problem that's in front of you that makes you makes makes it apparent that the model ain't fitting on a single device, you've got to spread it across many devices.
SPEAKER_01Then there are power and other constraints that are in front of us, right? We can't get power quickly enough, these data centers are large. And uh, you know, we didn't, I mean, we didn't start at this point, right? We if you look at if you go to you know rewind a little bit, Tensodyne got its uh was prior to this was named as Recognize, and we we had our CDSA funding in 2019, and we were more at that time the world was about image and image recognition and resnets and res net 54, res net 101.
SPEAKER_01And we started as an inference company, so we but uh because it was image recognition, we had to think, okay, how are we going to build uh a piece of silicon and a system that can do really exceptional performance deliver high performance for AI, for inference, for image recognition? And the market was automotive that we focused on. Automotive has a power constraint. The car can only give you so much energy. So there's a power constraint, and you have to deliver inference and you have to process, let's say, incoming frames of from cameras that are at eight megapixels.
SPEAKER_01So we set that up as a problem statement, and then the realization was that if you took traditional math and tried to do it in the car, you would blow the power budget of a car. You know, rather than using it to move the car more and more miles, you would basically you know shrink the range of the vehicle. So our approach to inference at that time was like, let's think about mathematically it being different. And let's approach math it from a first principled math perspective. And that's how we converge on using logarithmic math for AI.
SPEAKER_01So that set up uh if you know one has to be fortuitous, that set up the foundation for TensorTyne today. We didn't build a perfect device, but we uh we validated our math. We validated that that it really worked for AI. Uh and and the uh analogies were that, you know, first of all, logmath's been around since uh John Napier invented it over 400 years back. So it's not something that we dreamt of of. And uh log math has been used, you know.
SPEAKER_01If you think about like the Apollo program and all these programs, people used to use slide rules and logmat to determine escape velocity and re-entry and all these things. And then over the last two, three decades, we've been using logmat in DSPs. So it's not like we were the inventors of logmath, but our realization was that can we use logmath in AI? And so that that helped us, right, and set us up for today.
SPEAKER_00Gotcha. Okay, okay, interesting. So, for listeners who uh haven't heard of TensorDine before, TensorDyne actually started as a different company. What did you say the name was?
SPEAKER_01Recognize, R-E-C-O-G-N-I.
SPEAKER_00Recognize, okay, so probably image recognition, maybe it came from that. Yeah, yeah, yeah, yeah.
SPEAKER_01So exactly.
SPEAKER_00Um so you were looking at like convolutional neural networks, that kind of stuff, pre-LLM, pre-transformers. And you were asking, okay, how can we do this at the edge? Thinking automotive, so that you know, self-driving cars, uh, that kind of thing. Um makes a ton of sense. Um, and you obviously a power constraint there, which I'm sure we'll get to, it's the same power constraints in the data center, same but different, but like thinking about power. Um but you were you were taking a you were asking how can we do this incredibly efficiently?
SPEAKER_00Um, and the the idea was to use log math instead of the traditional like matrix multiplication, um, multiplies and adds. Like tell our listeners a little bit more about like what does log math unlock as the benefit.
SPEAKER_01So uh you don't you can't avoid matrix algebra. Matrix algebra is a fundamental for all of AI. So it's not like uh we could avoid that. But if you take a number and convert it to a log number, then uh and most of the operations are uh what are called mat muls, right? Which is basically, if you think about it, it's a multiplication, follow band accumulate, an addition. So we are doing trillions of multiplications and additions in AI. Uh, whether we're doing it in convolutional neural networks for image recognition or on LNMs, it's exactly the same.
SPEAKER_01So when you take a number and make it a log number, then log base to A times B is log A plus log B. So multiplier becomes an adder. And uh, you know, let's think of ourselves as young children. Let's say we are in elementary school and we're trying to multiply two numbers or add two numbers. Adding is easy, right? You just take the numbers and add them. But when you want to multiply, you know, you have to do this long multiplication and add a bunch of after that. It's the same thing in silicon. If you think about it, you have to have a lot more gates, you have to take a lot more area, a lot more power to multiply two numbers.
SPEAKER_01But you want to add two numbers, it's more easy. Uh adders are you know like a building block in silicon. So you saw the first part of the problem. The second problem part of the problem is really hard because now you have to take the the result of the multiplication and add it. And the minute you are in a logarithmic domain, adding becomes a problem. And if you were to take a traditional approach of a lookup table or tailor series, all the gains that you have in going to addition in place of multiplication are lost.
SPEAKER_01So our fundamental innovations that we patented was to figure out how to go back from a log to linear space and perform the additions without losing the gains in area and space and power that we got from multiplying the numbers, actually adding them, right? And so we spent inordinate amounts of time in mathematics. So though almost half the company spent almost half a decade in math, right? Because that's what you that's how uh you can be a revolutionary starter rather than an evolutionary startup.
SPEAKER_01Because then you can do because if you do it, if you do the same thing like everybody else does, and you have access to the same, say, silicon geometry, 7 nanometer, 5 nanometer, you're gonna have no advantages. You can't compete uh against the biggest guys. So you actually have to think about this differently, and that goes back to how you know, in my thinking, reflecting back in 96 and 97, how Pradeep and the other co-founders of Juniper thought about routers, right? And do it differently: forwarding in silicon versus forwarding with you know microprocessors.
SPEAKER_01So the similar analogy is applied here, and so we spend a lot of time thinking about that.
SPEAKER_00Nice. Okay, awesome. So you reach for log math because you realize, okay, we have a bunch of math moles to do, and if we instead of in multiplication is expensive from a power and a uh dice area perspective, um, and ultimately probably latency as well. And therefore, if we can convert it to the logarithmic domain, we can turn these into ads, but then eventually it has to get turned back, uh, converted back. And you guys thought really hard about how to do that uh efficiently, effectively, cleverly.
SPEAKER_00Um, and so okay, so then you said, okay, we figured this out. This is great, this is more power efficient, uh, small, fast. So then you took it to market ultimately, or at least built chips for automotive um to kind of prove out essentially the RD side of things. And to okay, now connect me from there to getting into data center inference.
SPEAKER_01All right, let's do that. Let's do that. So um think of a convolution neural network as like a three by three matrix. So you have to perform nine operations, nine multiplies, followed by all the additions that make add up those nine numbers. Uh so we had the building block for it, and uh we had functioning silicon in seven nanometer that was fully functional, and we were able to demonstrate taking camera streams and doing that. Um we hadn't incorporated a lot of the safety features that we needed for automotive.
SPEAKER_01And so at that time, in you know, uh late 22, uh early 23, Chat GPT happened. And we realized quickly that our building blocks were really set up perfectly for this moment. Uh, the market was 50 or 100 times larger. The ability to have uh a product from power on to get into revenues much, much faster in the data center, in in networking, rather than in automotive, because automotive has a long call cycle, right?
SPEAKER_01Yeah, it takes a long time to build a car, you gotta certify it, you gotta make sure it's safe. There's a number of things that need to happen. So uh we coalesced as a team and said, let us look at our technology and see how applicable is it for LLMs. An LLM, you do one by one multiplication, not nine, three by three. Great. Uh well, LLMs, so if you look at ResNets and all, think you know, 100 million parameter systems, and now you're going from 100 million to multiple hundred billion to trillion parameters.
SPEAKER_01So the memory hierarchy, the memory architecture has to change. You have to think of memory slightly differently. Yeah, uh, but your fundamental building blocks for math you can build upon. And so that's what we did. And so we started architecting in 2023. So we had the good fortune of retrospective view on um on math uh and 100 million parameters, but we also had a view that what was a problem statement in front of us? That is that you could go from multiple hundreds of billions to maybe trillions of parameters, and so how do we build an architect a chip for that?
SPEAKER_01We still hadn't solved the networking problem though. So we said, okay, let's understand how to build first an accelerator that could be world-class and deliver world-class performance for LLMs, and then the multi-hundred billion parameter problem now became a multi-trillion parameter problem. Yeah. And so now you have to think about the networking problem next.
SPEAKER_00Yes, yes. Okay, so you you guys said, oh, we've got all the building blocks, the opportunity is enormous, uh, the path to production is shorter. Uh I'm sure it was hard decision, but in retrospect, it's kind of a no-brainer. Um, but of course, you had to make new memory dis hierarchy decisions. And then uh to your point, that leads to the like, oh, that the the model weights are not gonna fit on one chip. Therefore, we have to connect lots of chips, therefore, networking has to be like a first class design priority.
SPEAKER_01It does.
SPEAKER_00Yeah, so tell us more then about your networking and where do you guys end it up?
SPEAKER_01Yeah, so now now um now let's for a moment step back and think about the first principles of uh how do you build uh JAI inference systems, right? So this compute, uh there's constraints that are there in place, right? What is your compute? Whether you compute bound, uh whether you're like memory bandwidth bound, how much bandwidth do you have coming in from external memory, memory capacity bound, how much memory can you put? Right? And then when you think about the networking problem, then you say, okay, how much bandwidth do I have facing uh in and out of a let's say scale-up network?
SPEAKER_01And then what is my scale-up network latency? Because that also might have an effect. So now you think about silicon and you say, okay, how much of my real estate do I allocate for all of these things? How much do I allocate for compute? How much do I allocate for memory interfaces and memory capacity? Uh how much do I allocate for own chip memory? That's S RAM, and then how much do I allocate in terms of Interfaces and bandwidth and for my fabric interfaces.
SPEAKER_01So when we sat back and looked at it, and this was like I think I think I believe in the summer of 2023, the realization from some of me and some of my colleagues were gosh darn it, we'd been building these networking systems for 15, 20 years at Juniper networks. And I had the good fortune of leading some other efforts at Juniper for in silicon uh almost 17 years. And in the back side of a router, in the rear of a router, is a scalp fabric.
SPEAKER_01Because if you think about a router, it's got lots of front side ports. So think of like a high-end router, it could have, I don't know, almost over hundreds of 800 gig ports. So traffic comes in from one port and then goes into the back, the rear of a router, and somehow that traffic has to be routed to an outbound port.
SPEAKER_02Yeah.
SPEAKER_01So a router typically is you know in a large um point of presence ATT or Verizon or connects big data centers of Google or Amazon. So in the rear of a router is a scale-up fabric. And it's designed to deal with any to any traffic, any packet size. So typically, you know, route in typical, if you look at TCP IP, you could have like a small 40-byte, 20-byte packet, you could have a jumbo frame, 9K, and this and the fabric has to deal with this random traffic.
SPEAKER_01Uh, it might have like email, which is SMTP traffic, or it might have real-time video, like this call. So the router has to be robust under all conditions. And so the realization internally was like there is a ready-built scale of fabric available for us if you can make the connection. And so we had the good fortune to be juniper alum and HP Juniper now, HP Juniper Alum. And so we reached out and partnered with Junpo Networks and leveraged one of their highest-end routers and took the scale of fabric in the rear of it and leveraged that for our system.
SPEAKER_01So suddenly we went from having an accelerator that could be world-class, highest performing, to an accelerator that is now well connected in a scale-up fabric. So this combination of networking and compute, if you had that, if you had the background, if you thought about it, then some of these docs get connected. And we were fortunate to do that.
SPEAKER_00Gotcha. Fascinating. Okay, so you had the compute, and then you needed ultimately you needed a scale-up fabric because you're like, hey, we need to connect, let's say, 72 chips together into a rack. And instead of having to start from scratch there, you're able to partner with HPE Juniper uh and come up with a scale-up fabric that that fits your needs there. And okay, so talk to us more about the scale-up fabric because ultimately I know that this could limit like the amount of tokens per second at the end of the day.
SPEAKER_00Um, so like but but but but take us there and explain it.
SPEAKER_01Let's think about um the system broadly, right? Um so I I've always been, I mean, I think I've had good fortune to have mentors through my career that have told me that when you design systems, think about the balance. So if you if you over-pivot on one resource, let's say you're compute, compute heavy, then you might be in points in time where the computer is sitting idle. Uh if you are uh if you have undersized something, then uh you might be starved of that resource.
SPEAKER_01So you have to think about how uh you allocate this precious resource in a in a silicon. So uh the thing that happened to us was that we wanted to build a scale-up fabric that was ready for yesterday, today, and tomorrow. And the best scale-up fabric in the world was on the backside of a router. Uh, why does why is that? Because it has to have some really exceptional any-to-any characteristics.
SPEAKER_01So you can connect like multiple accelerators or multiple, think of routing devices and you can connect them together. It had to have incredibly low latency because we know that you know, on a video call like this, latency has an effect on how we experience each other. Uh, we we you know today we use FaceTime and or whatever, and we call people on the other side of the globe. Think of that latency, right? Uh with multiple hops. Uh, it had to be congestion free, it had to be reliable, and it had to be it had the capacity, the fabric or the scale-up network had the cap should have the capability that you can push its utilization all the way up to 99 plus percent, and it will still perform really well under those conditions, under loaded conditions.
SPEAKER_01So, all of these were key elements for us. If you had to look at all of these elements, that was the the Juniper Fab HP Juniper Network cell fabric in the back side of a router had those properties. So its latency characteristics are like sub one between one and two microseconds. Okay, so now you look at this picture and you say, wow, this is the this is the perfect fabric for us. What happened in parallel in the world in networking, in sorry, in AI models, the models got larger and the traffic patterns went from rhythmic tensor parallelism kind of uh patterns to random traffic because when you do experts, the experts are random in nature.
SPEAKER_01So now communication uh at the if uh for high tokens per second per whatever, communication now becomes a bottleneck. And if your fabric is better than anybody else's, that communication bottleneck is uh um relieved somebody.
SPEAKER_00Sure, sure. Okay, this is so interesting. So taking a step back, you guys said, hey, uh when you when you first started thinking about this, you needed to balance um compute, memory, capacity, memory, bandwidth, networking bandwidth. And on the uh scale up bandwidth, you said essentially, yeah, what's like the the best maybe maybe you're able to look forward and think that this would be like the defining bottleneck of sorts. Um what's like the best possible scale-up networking technology that's out there?
SPEAKER_00And you looked and you said, hey, at the back of these world-class routers, like that's great. And to your point, they can handle you know heavy load, low latency, and so on and so forth. Um then so you so you're adopting that. And now to your point, we're in a world where it's like, yes, you actually do need a rack of you know, 72 chips, maybe many racks, even running a 2 trillion parameter, a mixture of experts. So the experts are living in different places, and for every token, it might need to go to a different expert. So uh irregular communication pattern.
SPEAKER_00Um, and now contrast it for listeners with maybe like what's being used today, like when you say one microsecond, um, how should they benchmark that against like um you know, let's say in an NV Link or AMD's UA link or something like is it like an order of magnitude, or how should we think about it?
SPEAKER_01Yeah, so uh let's reflect back uh the last couple of years. So the first scale-up system for AI in the world is a GB200 NV Link NBL72. So it's the first time because prior to that, uh only eight devices got connected together. Yeah, so to connect 70 devices, it's gen 1. NV Link 72 is gen 1. We are in gen 7 of a scale-up fabric when we leverage leverage the HPE Juniper Networks fabric. That's point number one.
SPEAKER_01Uh so there's a learning cycle, right? Because when you build scale up systems, you have to build them, learn them, refine them, you know, this iterated process. Yeah, that learning process we can leverage uh from HPE Juniper. Uh so that's one part. Now, if you look at the world of AI, most of the car most of the vendors now said, okay, we need a scale of fabric. So you saw uh consortiums come together with UA Link, uh with ESON, still early days, they're still in you know specs and then early implementation.
SPEAKER_01We still don't have accelerators that can talk UA Link or and and UA Link fabric chips and all of those things. So there's a learning cycle that has to we skip those learning cycles with with Juniper because we know the architecture of the uh of that fabric. Uh, a lot of our engineers were like core chief architects from some of that in the prior generation. So we know that. That's one aspect of it. If you look at the Helios system from AMD, they they needed to get to this quickly. So they are now tunneling UA link kind of uh packets through a standard Ethernet uh kind of network.
SPEAKER_01Uh so now you think about, you know, typically when you move data over uh any link, you have to think about okay, how much space is used by headers, addresses, and all this other stuff, and then how much is payload? Because what matters is how much payload and how long does it take to transmit the payload. So all of these things uh are important elements in our choices. The other thing is, as a startup, uh if you carry the burden of building silicon, but also carry the burden of building a system, uh, you have to qualify the system, the system has to have reliability characteristics, there's all of these other things that now affect your time to market.
SPEAKER_01So by partnering with HP Juniper Networks, we can leverage an existing shipping system that's shipping today from Flexus Factories in Penang. So we get to get to uh we get to take uh our silicon, integrate it into and transform a router into a compute box and get to market faster with a reliable scale of fabric. So I honestly believe, hand on heart, that NV Link is the first, NBL72 is the first scale of fabric in the market.
SPEAKER_01We will be the second vendor in the market with a true scale of fabric, right? Uh that has exceptional latency characteristics. Now, what is the latency? It's between one to two microseconds. And this is, by the way, I don't know for a fact, but this is what I've heard anecdotally that typically uh the NB Link fabric has some latency characteristics that are not amenable to, for example, decode performance. So maybe there uh there's some refinement that needs to happen with NBL72 in future generations.
SPEAKER_01So Netnet, as a result of that, we end up with a system that's really well balanced and can do pre-fill really well, but can do decode really well too. So you don't have to have uh you know this heterogeneous systems that have to appear to solve this high, what's called interactive uh kind of AI needs of tools.
SPEAKER_00Yes, yes. Okay, great. Let's go into that thread too. So there was the era where it was just like use you know hoppers for inference and then Grace Blackwells for inference, and then um some early AI ASIC startups who started you know well before LLMs, um, namely Grok and Cerebris, had S SRAM heavy um designs that worked in the inference world for high interactivity, decode very fast. Ultimately, of course, they had their own trade-offs of uh you don't have HBM, you have to chain together tons of these to get enough memory.
SPEAKER_00Um and then recently in the last you know 12 months, basically, we've lived in this world where it's like, okay, if you want really high interactivity, you can use an NVIDIA system with Grok. Um, and that lets you extend that Pareto curve to the very high interactivity, low throughput, but high interactivity. Um and you know, we've seen AMD uh partner with Cerebrus. Um, AMD also bought uh TALUS recently, which is a different conversation. But um, so there's we're kind of in like phase two. Phase one was just run everything on GPU, GPU.
SPEAKER_00Phase two is disaggregation, split, pre-fill, and decode, put pre-fill on the GPUs, put the decode on the SRAM heavy chips. Um and then you're kind of hinting at like this next era beyond that, where I, you know, I think TensorDyne is aiming, and and so are some other AI ASIC startups, um, which would be actually can you just use one uh chip to do both workloads, pre-fill and decode? And so so, but tell us more about that because it feels a little counterintuitive because because uh you know, as you said, you know, pre-fill may be compute bound, decode is memory bandwidth bound.
SPEAKER_00So, how does like one chip do do both?
SPEAKER_01Really good question there, Austin. So um so let's uh go back maybe 10 months back. So something happened towards the end of last year, right? Uh GPD 4.5 got really good, and then suddenly people started seeing wow, the core is generating is functional. I I don't have to debug it a lot. And then comes uh no GPD, I didn't 5.3, I correct me wrong, and then uh Opus 4.5, right?
SPEAKER_01And uh so suddenly the demand for AI goes through the roof. And what are those models? My get my gut says they're over 2 trillion parameter models. So there's a realization in the industry that those models cannot really work at a high in high interactive environments where the developers are saying, listen, I need code faster, I need everything faster. And so the reaction to the industry is well, we have great systems for training, they do pre-fill really well, and you know, uh kudos to the GROX and Cerebressel world that had only shown for smaller models like Lama 70 or GPT OSS 120B, high tokens per second per user.
SPEAKER_01So now comes the problem statement. Like I have a system that does really well in pre-fill, can be used for training, but now let's talk about inference only. Now I need to uh pair it with uh a system that has a lot of SRAM and has low latency and I can do decode faster. So I think that is an intermediate solution from these vendors, whether it's Nvidia plus Grok or Amazon Trainium plus Cerebras or Helios plus Cerebras. Uh imagine you know nine racks or 14 racks to run a 2 trillion parameter model.
SPEAKER_01Imagine trying to build a compiler that compiles in CUDA for a for a Vera Rubin and not CUDA for a Grok system. And then imagine connecting them with lots of Ethernet switches because you have to hook up these chassis together. So um that is an answer to the demand of the market, right? Because the demand from everybody else is that we want to run these two plus trillion parameter models, and now if you look at Kimi K3, it's 2.8 trillion parameters. Uh, but we need we need high token, you know, in the agentic world, you want high token rates, for example, tokens per second per user.
SPEAKER_01Here, the user could be a human or could be an agent for all you care. All right, so now let's get back to what did we do at Encityne? So when I look at Silicon Real Estate, there's only so much space. Now, if my compute takes less space, what does it free up? It frees up space for more SRAM. So now you start imagine now a chip that has more SRAM, kind of like the grox and the cerebrus of the world, but also has HBM and has the compute capacity of an Nvidia or an AMD.
SPEAKER_01So you get the best of both worlds. You get what's called uh, you know, you can do high throughput, but you can also do high tokens per second per user uh with the same system. Uh why? Because ultimately what you want to do is when you have compute, what matters is uh model flops utilization, MFU, for example. Well, if you have a lot of SRAM adjacent to, let's say, your compute array, you can keep that compute array really busy. And you can hide the latency of getting data from HBM.
SPEAKER_01So if you have a lot of SRAM, you are not penalizing yourself by stalling your compute array every time you had to go and fetch data or activations or weights from HVM. So we start getting the kind of unique properties of both of the best of those both worlds. But that's necessary but not sufficient. Because now suddenly there's communication now that now these models also have. So, you know, you got to compute, then you got to communicate, then you got to compute, then you got to communicate. And this rhythm of compute, communicate, or expert selection now, the communication starts becoming a dominant effect on latency or through throughput or tokens per second per user.
SPEAKER_01So our good fortune was to stick with log math, have the power characteristics, not blow the silicon, you know, you don't have to build a chip that's reticle sized to achieve the performance uh that that's in par with others. And now suddenly you get the benefits of both of these. So now you have a great machine for pre-fill and decode.
SPEAKER_00Ah, yeah, yes, yes. Okay, so the key insight is by using the logarithmic Mac Math that we talked about earlier, the compute doesn't need to be reticle size, it can be smaller. So then the question is, what can you do with the rest of the silicon? And oh, we can have even more SRAM, for example. So you can have a lot of compute, a lot of flops, you can have a lot of SRAM, which is great for D code, um, but then you can still also have HBM for pre-fill and just having HBM in general, um KV cache, that kind of thing. Um and then you also feel confident in the competitiveness of your scale-up network.
SPEAKER_00So as you're communicating between all these things, you feel competitive there too. So ultimately then, um, if I have a rack of Tensordyne, oh, which, and by the way, we'll touch on power too. But my first question is if I have a rack of Tensordyne chips, um, it is it you know 72 chips in your rack, and if so, like do you still sort of split pre-fill and decode amongst them within the rack?
SPEAKER_01So you have immense flexibility. So, first of all, we are building the Tensordyne system, it's only 13 rack units. So it's a tiny, it's one fourth of a rack. We can put four of them in a rack. At one fourth of a rack, think of our system, it has 72 chips in one fourth of a rack. That's point number one. Secondly, it consumes only 30 kilowatts. So if you were to compare do apples to apples, the comparable system is like a Blackwell GB300.
SPEAKER_01So one fourth of a rack, one full rack. 30 kilowatts, 150 kilowatts. So how did we do that? So, first of all, we leverage a Juniper air cooled system, so it's not liquid-cooled. Secondly, we have a scale of fabric that has the characteristics that are necessary for these multi-trillion parameter MOE models. Thirdly, our log math allows us to have silicon that consumes much lower power. Yeah, and so you get you get you know, you get multiple benefits that are compounding in nature that now allows us at our uh you know to have performance at the you know GB300 blackwell levels in a one-fourth of a rack.
SPEAKER_01The second thing is that because we have the scale of fabric as this elegant properties, you can you can do you can put some chips on pre-fill, some chips on decode. Or in our full rack, we can basically say, okay, there's one of our pars is a pre-fill board, and three pars are decode pods. But ultimately, if you look at it, because of our characteristic nature of our any-to-any scale-up fabric and the characteristic nature of keeping uh keeping a very high MFU because the SRAM keeps that, you know, I always refer to it as the compute dragon, keep the dragon fed because the dragon's hungry, you've got to keep it, right?
SPEAKER_01You can't you can't let it off. So you keep the dragon fed. Now you get a system that is so much more balanced and can perform really well. And in some of our metrics, now we are yet too power on our system, but in some of our sense, we think that we could have advantages in the order of almost a magnitude over the latest, greatest systems in the world.
SPEAKER_00Wow. Wow. Okay, I'm gonna reflect this back because it feels almost too good to be true. Um so you can fit 72 in a quarter of a rack, that's only 30 kilowatts. You can stack four of those in a rack, um, so 120 kilowatts uh or so, and then it that whole rack is air-cooled, right? Yes, yes, yes. So you could deploy it in brownfield data centers, existing data centers.
SPEAKER_01Yeah, so imagine, you know, if you look at it, we have data centers around the world, you know, some of the hyperscalers have 400 data centers around the planet. You they're not liquid-cooled, you can't retrofit them, right? They're in some of them are in bigger cities or in countries where you don't have enough power. But you want to get AI around the globe. How are you going to do that? So our system is because it's a telco rack, it's 19 inch wide, it fits in AT ⁇ T, Verizon, Deutsche Telecom, NTT, New York Stock Exchange, that rack is perfect for being used in anywhere in the world.
SPEAKER_02Yeah.
SPEAKER_01So suddenly you can get that benefit. The other is that you can imagine building smaller data centers, but having the performance characteristics of a large data center. So imagine, you know, you know, with our system, you can actually Actually, get you might need you can get the you can get like one eighth the power you can get uh you know, you can have a data center that's similar to a very large data center uh with the latest systems from our biggest competitors. Yeah, so now you can deploy them in smaller locations, smaller cities, you can put you know 20, 30 kilowatt kind of data centers not needing a gigawatt data center, but having the performance characteristics of a 200 megawatt or 500 megawatt data center.
SPEAKER_01So you get all of these benefits, too good to be true, but you know, if you build systems, you know, you you have to know the system characteristics and system performance before you build it. You don't want it discovered in the lab. So yes.
SPEAKER_00So we've spent a lot of time at this simulating everything. Totally. Okay, very interesting. So then my mind naturally goes to a couple questions. One, then who are the target customers? Because on the one hand, I think of like enterprise, um, because it can fit into existing enterprise data centers, and maybe instead of needing to buy, you know, nine racks or 13 or however many, you know, maybe they can buy one for their needs or or a few or whatever. Um, and then two, yeah, timing. So um, you know, I think you had mentioned that this isn't totally taped out or stood up or productionized or something.
SPEAKER_00So yet remind us like, so tell us who are your target customers? Because of course I can also see, yeah, hyperscalers in the model labs and and and stuff being just as interested. Um, and so maybe like target customers' route to market and timing.
SPEAKER_01Yeah, so um our target customers are certainly number one hyperscalers, number two, uh, a lot of the neo clouds. We have a significant number of letters of intent from almost all the neo clouds in the world. Uh well, I want to clarify neo clouds in uh North America and Europe. Um and and then uh maybe some sovereigns and some enterprises, right? So, but as a startup, we have to be laser focused, right? We don't have a very large sales team, so we have to focus on um a targeted set of customers, and we are doing that.
SPEAKER_01Um the second question was timing, right? Yeah, yeah, yeah. Uh so we've taped out our three-nanometer chip uh with our uh awesome partner Broadcom. And so we expect devices back uh late October, early November, and we expect to power on our system and uh and then you know, probably sometime in Q1 start doing beta with customers. We have some customers who have signed up for beta, and then potentially go into production, early production, sometime in end of Q2, early Q3 next year.
SPEAKER_01So we're not far away from there. Now, how can we do that? Because 80% of the system is already validated, uh, qualified, certified in like 70 countries. We are building it out of the same factory. Uh so uh uh in in Penang, Malaysia, so we can actually quickly go from power on to qualification to certification to shipping uh of our system. So we have our worker out uh for the next, I would say, 10 months, but uh we're fairly confident that we can get from here to there pretty rapidly.
SPEAKER_00Nice, nice. So is that because you're riding the supply chain that Juniper was already riding, or is this the advantage of partnering with Broadcom or both? It's both.
SPEAKER_01So, you know, Broadcom is one of the largest consumers of TSMC capacity, both TSMC and HBM capacity, right? They have a deep partnership, and we have an exceptional relationship with the Broadcom team. They're great friends and great supporters of us, and uh and so we designed uh of course our front-end design is ours, but the physical design and the partnership with TSMC is uh Broadcom-led. Uh so that's one part. So that means that from a supply chain perspective, as long as we forecast uh well in advance, we can get uh parts from Broadcom, right?
SPEAKER_01That's part number one. Uh qualified tested parts. The second part of the story is that can we get to volume with our systems? And this is where the partnership with HPE Juli networks and the fact that things are shipping out of a flex factory, and flex knows how to build these systems and volume, gives us those advantages. The third most important part is that these systems have to be reliable because you're running business critical AI now in there. So uh Juniper's routers and HP Juniper's routers are carrier grade.
SPEAKER_01There's something called finines above time. So finines above time, if one were to Wikipedia, right, is that the system cannot be down for more than five point some odd minutes in a year or 680 milliseconds in a day. So that is a reliability characteristic that we leverage uh from our partner. Uh, and so we are able to, we will be able to deliver systems that are reliable uh and can run the largest models in the world, right? At 2.8 trillion parameter, like a Kimi K3 or a Quen or uh from the Frontier Labs, their biggest models, right?
SPEAKER_01Uh our systems have an adequate amount of memory. You know, if I if I look at uh Quarter Rack system, we have we have 10.8 terabytes of HPM capacity. We have uh you know significant, I would say over 18 gigabytes of SRAM. So imagine we have SRAM capacity of the the big S RAM guys, and we have DRAM capacity of also the largest thing. So now uh you know, whether it's KV cash and agentic workflows, we we are well set up for those systems, uh for those markets and those customers.
SPEAKER_00Fascinating. Amazing. A really nice way to punch above your weight, so to speak, as a startup by kind of by you know partnering with people who can help bring you along. Um when you mentioned flex, by the way, is that flextronics? It's flextronics. Yeah, yeah.
SPEAKER_01Okay, yeah. And and you know, I mean, uh most of Silicon Valley thrives on partnerships, right? You can never go alone, right? Whether it's silicon partners, system vendor partners, uh the ecosystem actually lends itself to that partnership. And we we uh you know, we benefit from it, uh, we mutually benefit from it, and we we really love these partners because they are helping us get to that scale, will help us get to the scale, and they're very supportive of us, right? They're supportive of a startup.
SPEAKER_01Not only that, HP Juniper Networks is an investor in the company. So you think about it, they uh they, you know, in in all of these things, you know, you can never go it alone. It's like building a team, right? In the company, the team is multidisciplinary, but you can't build a product by yourself. All these functions come together. It's the same in the ecosystem of uh of vendors and partners who come together to build these uh solutions. Yeah, sure, that makes sense. And customers are the final you know arbiter of product, right?
SPEAKER_01And we we customer validation is gonna be super important for us over the next many months.
SPEAKER_00Yes, totally. Okay, so let's let's land the plane there, customers. So, you know, you mentioned uh really ramping up in 2027 with customers, beta, and then production. Um, the one thing we hadn't touched on yet was software. So, you know, for those model labs, those hyperscalers, what does it look like as far as how heavy of a lift is it to take you know your system and get their software, uh their frontier grade software up and running?
SPEAKER_01Yeah, uh exceptional question, uh Austin. I think a few things have happened, right? One is that from the signals of these disaggregated stories, we understand that the morts of CUDA don't exist anymore for inference because we are proof positive of a number of inference companies running solutions, and then these disaggregated solutions with CUDA and non-CUDA or neuron and non-neuron or rock em and non-ROKEM, right? So we have this.
SPEAKER_01So uh I think there are a couple of things that have helped us. Uh, one is obviously we have to build a robust compiler for our hardware, uh, but also at the higher layers, having a technology that can take PyTorch or Triton kind of models and then convert them quickly and compile them quickly is super modern to spend a significant amount of time. So, for example, in our engineering organization, almost 60 to 70, 60 plus percent is software, right? Because you have to have a software team that does that. But one of the things has happened with this beauty of uh these new Opus 4.5 and and uh you know GPT 5.6 and other models is that then agentic workflows are now proving that you can take kernels and quickly get them ready for your hardware.
SPEAKER_01Yeah, so we have incredible momentum on taking like the best models in the world and quickly getting them running on our, you know, in our simulators and then eventually in our hardware. So the the walls for taking new architectures and new systems are quickly crumbling when it comes to software because you can take models and because you have all these workflows and automated kernel generation capabilities, that you can actually get to there, get to that, and also get to high what's called utilization of the systems quickly.
SPEAKER_01That means that you can get first proof positive that the model works on the system, and then you can iterate quickly to get the model performance up in the system. So both those things seem to coincide with uh, you know, our systems coming on.
SPEAKER_00Yeah, yeah. What yeah, what fascinating timing um with the rise of the agentic AI to kind of help you with that last hurdle that people are a little concerned about, you know, am I gonna have to spend a lot of time, not to your point, not only getting my code to run on this new system, but also optimizing it to get the most utilization out of it. Indeed, indeed.
SPEAKER_01And that is super important because you know you don't want an 800 horsepower car and get like 50 horsepower earth, right? Yes, because then you're leaving stuff on the table, right? You're paying for it, but you're not using it. Right. And we believe that we have we will have some of the best uh MFUs in the industry with our system.
SPEAKER_00Nice, amazing. Okay, well, we are at time, folks. I hope you learned a lot about RK and Tensordyne and networking, uh, a lot in logarithmic map. We covered a lot of ground here. Um, RK, you'll just have to come back and keep us updated, you know, as you guys get closer to uh delivering ships and uh chips and systems and when you're cut you have customers to talk about, you know, can't wait to hear more.
SPEAKER_01Austin, again, thank you for the opportunity. Thank you for uh you know the very thoughtful questions uh and you know, and navigating all the different parts of the terrain uh on our behalf. And you know, uh we'll certainly keep you in the loop, we'll keep you posted as we make progress, as we bring up our systems, as we get to beta, and um would love to come back and tell you about our success and our progress. Uh again, like I want to actually give credit to my broader Tensordyne family because we we won't be here without the the complete dedication of a number of people, and then most importantly, thank our partners, Broadcom, TSMC, uh, HP Juniper Networks, and Flex and others, because we can't be here without any without all of them helping us in this journey.
SPEAKER_01So thank you, Austin. I really appreciate it.
SPEAKER_00Yeah, team effort, very cool. All right, we'll talk, we'll talk next time. We certainly will. And my best to you, Austin. Thank you. Thank you. Cheers.
Austin Lyons and Vik Sekar break down Google's two new TPU systems, the 8i Max and 8TX. These are not new silicon but new system configurations of the existing V8 chip, designed for specific workloads. The conversation...
Jensen Huang resigned as Nvidia CEO this morning, and Austin and Vik recorded two hours after the news broke. They argue that a board must prioritize corporate integrity over short-term shareholder value, even for a...
Intel just had its worst day on the stock market in 55 years, with shares falling nearly 30%. But here's the strange part: Intel actually reported a strong quarter. Revenue grew, its data center business surged, and the...
The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet...
Marvell's potential acquisition of Lumentum is a strategic vertical integration play to capture the massive value optical components now unlock in the AI era. Andrew Schmitt and Scott Wilkinson of market research firm...