Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DEV

DEV artwork

Why Your AI Is Slower Than a 1998 Modem — And How to Fix It

Aug 4, 20268 minAudio

A model that aces every benchmark but makes users wait five seconds for a response isn't ready for production - it's a liability. This episode of Development tackles one of the most common and costly gaps in modern AI deployment: the difference between a transformer model that works and one that works fast enough. Drawing from this in-depth guide on optimizing transformer models for low-latency inference, the episode walks through why inference is so expensive by design, and what engineering teams can realistically do about it. Here's what the episode covers: Why transformers are inherently compute-hungry - every token generated requires loading and applying hundreds of millions (or billions) of parameters, creating a latency problem baked into the architecture itself. The quadratic cost of self-attention - why longer inputs don't just take a little more time, they can take dramatically more, and what that means for real-world use cases like extended conversations or long documents. Model compression strategies - a clear-eyed look at quantization (including quantization-aware training), pruning, and knowledge distillation, with honest assessments of where each technique can backfire. Hardware choices and their trade-offs - from GPUs and TPUs to dedicated AI accelerators, why running transformer inference on the wrong hardware is a silent performance killer, and how toolkits like TensorRT can help close the gap. Serving infrastructure and deployment models - how frameworks like NVIDIA Triton and ONNX Runtime enable dynamic batching and smarter scheduling, plus the latency trade-offs between cloud, edge, and on-premise inference. Treating optimization as an ongoing discipline - why a one-time tuning sprint isn't enough, and how profiling tools and a layered approach to trade-offs separate teams that stay fast from those that fall behind. The episode makes a compelling case that speed and accuracy aren't opposing goals - they're both engineering problems, and both deserve the same rigor. The strategies discussed here aren't theoretical; they're the same techniques production teams are applying right now to close the gap between a model that impresses in a notebook and one that holds up under real user load. More from the show: if you're interested in customizing models for specific domains, check out the episode on Fine-Tuning LLaMA 3 With LoRA: Making AI Work for Your World. DEV

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. The U.S. Has More Data Centers Than the Next 14 Countries Combined

    Nov 20, 20265 minAudio

    The global data center map looks lopsided by any measure - but the full picture is more striking than a simple site count suggests. This episode of DEV.co digs into a newly published graphic analysis showing just how...

    Credit the people on this episode

  2. From $30 Million to $5 Billion: The AI App Explosion by the Numbers

    Nov 18, 20265 minAudio

    The generative AI app market has posted one of the most dramatic growth curves in consumer software history - and the numbers behind that curve reveal a story far more nuanced than the headline figure suggests. This...

    Credit the people on this episode

  3. AI Adoption: Why the Numbers Are So Far Apart

    Nov 15, 20265 minAudio

    A single statistic can make AI feel like a done deal - nearly every organization already on board, the window to act almost closed. But a competing figure from the U.S. Census Bureau tells a starkly different story....

    Credit the people on this episode

  4. 83% of Developers Use AI Tools — But Only 6.6% Actually Trust Them

    Nov 13, 20265 minAudio

    Widespread AI adoption among software developers is no longer a trend to watch - it's the current reality. But a sweeping Stack Overflow survey of nearly 31,000 developers reveals a striking contradiction at the heart...

    Credit the people on this episode

  5. Private RAG Over Microsoft 365: Permissions, Delta Queries, and What Actually Ships

    Nov 11, 20265 minAudio

    Enterprise teams keep hitting the same wall when grounding LLMs on SharePoint and OneDrive: a working prototype that fails the moment security asks who actually owns access to the returned content. This episode of...

    Credit the people on this episode