Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Semi Doped

Semi Doped artwork

Can Pre-GPT AI Accelerators Handle Long Context Workloads?

Jan 26, 202638 minAudio
ShareXLinkedInFacebookWhatsAppEmail

OpenAI's partnership with Cerebras and Nvidia's announcement of context memory storage raises a fundamental question: as agentic AI demands long sessions with massive context windows, can SRAM-based accelerators designed before the LLM era keep up - or will they converge with GPUs? Key Takeaways 1. Context is the new bottleneck. As agentic workloads demand long sessions with massive codebases, storing and retrieving KV cache efficiently becomes critical. 2. There's no one-size-fits-all. Sachin Khatti's (OpenAI, ex-Intel) signals a shift toward heterogeneous compute - matching specific accelerators to specific workloads. 3. Cerebras has 44GB of SRAM per wafer - orders of magnitude more than typical chips - but the question remains: where does the KV cache go for long context? 4. Pre-GPT accelerators may converge toward GPUs. If they need to add HBM or external memory for long context, some of their differentiation erodes. 5. Post-GPT accelerators (Etched, MatX) are the ones to watch. Designed specifically for transformer inference, they may solve the KV cache problem from first principles. Chapters - 00:00 - Intro - 01:20 - What is context memory storage? - 03:30 - When Claude runs out of context - 06:00 - Tokens, attention, and the KV cache explained - 09:07 - The AI memory hierarchy: HBM → DRAM → SSD → network storage - 12:53 - Nvidia's G1/G2/G3 tiers and the missing G0 (SRAM) - 14:35 - Bluefield DPUs and GPU Direct Storage - 15:53 - Token economics: cache hits vs misses - 20:03 - OpenAI + Cerebras: 750 megawatts for faster Codex - 21:29 - Why Cerebras built a wafer-scale engine - 25:07 - 44GB SRAM and running Llama 70B on four wafers - 25:55 - Sachin Khatti on heterogeneous compute strategy - 31:43 - The big question: where does Cerebras store KV cache? - 34:11 - If SRAM offloads to HBM, does it lose its edge? - 35:40 - Pre-GPT vs Post-GPT accelerators - 36:51 - Etched raises $500M at $5B valuation - 38:48 - Wrap up

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Google Built Two New TPUs Without New Silicon: The 8i Max and 8TX

    Nov 17, 202654 minAudio

    Austin Lyons and Vik Sekar break down Google's two new TPU systems, the 8i Max and 8TX. These are not new silicon but new system configurations of the existing V8 chip, designed for specific workloads. The conversation...

    Credit the people on this episode

  2. Jensen Huang Resigns

    Nov 12, 202632 minAudio

    Jensen Huang resigned as Nvidia CEO this morning, and Austin and Vik recorded two hours after the news broke. They argue that a board must prioritize corporate integrity over short-term shareholder value, even for a...

    Credit the people on this episode

  3. Intel's Earnings Disaster: Bad Luck, Bad Planning, or Both?

    Nov 9, 202657 minAudio

    Intel just had its worst day on the stock market in 55 years, with shares falling nearly 30%. But here's the strange part: Intel actually reported a strong quarter. Revenue grew, its data center business surged, and the...

    Credit the people on this episode

  4. 3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

    Nov 5, 20261 hr 16 minAudio

    The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet...

    Credit the people on this episode

  5. Why Marvell Wants Lumentum: Vertical Integration in AI Optics With Cignal AI

    Oct 30, 202640 minAudio

    Marvell's potential acquisition of Lumentum is a strategic vertical integration play to capture the massive value optical components now unlock in the AI era. Andrew Schmitt and Scott Wilkinson of market research firm...

    Credit the people on this episode