Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DEV

DEV artwork

Why Deploying LLMs on Serverless Is a Beautiful Disaster

Jul 26, 20268 minAudio

Serverless computing promises infinite scale, zero infrastructure management, and pay-per-use economics - so the idea of running a large language model on top of it is understandably tempting. But the gap between that pitch and production reality turns out to be enormous. This episode of Development unpacks the very real architectural friction explored in this deep-dive on deploying large language models in serverless environments, walking through the core challenges and the hybrid strategies that actually hold up at scale. The conversation covers a lot of ground for engineers weighing up this architectural choice: Cold starts as a dealbreaker: When a serverless function wakes from idle, loading a multi-gigabyte LLM before processing a single token introduces multi-second delays that make real-time applications essentially non-functional - and "warm instance" workarounds quietly abandon the serverless cost model entirely. The statelessness mismatch: Serverless is built around clean-slate, stateless invocations, while LLMs depend on persistent conversational context. Offloading state to external caches or databases solves this in theory, but adds latency, new failure modes, and significant debugging complexity. Hard compute ceilings: Platform limits like AWS Lambda's 10 GB RAM and 15-minute execution cap are nowhere near sufficient for frontier-scale models. Teams are forced to choose between aggressive quantization or offloading inference to GPU-backed services - either way, the pure serverless architecture doesn't survive contact with the model. Storage and loading overhead: With no persistent in-memory model between invocations, every cold start may require downloading gigabytes from object storage, while persistent file system alternatives add their own cost and complexity. Security and isolation concerns: Ephemeral function instances handling sensitive user data, pulling open-source dependencies at runtime, and processing prompts vulnerable to injection attacks create a security surface that demands far more discipline than serverless's simplicity tends to encourage. The cost trap: Per-invocation billing sounds economical until LLM request volumes scale up. Compute-heavy, slow-generating models can turn a seemingly lean architecture into a budget crisis - making the hybrid model not just pragmatic, but financially necessary. The episode lands on a clear verdict: the teams that succeed with AI infrastructure aren't the ones chasing architectural purity. A hybrid approach - serverless handling lightweight orchestration, routing, and preprocessing, while dedicated GPU-backed infrastructure takes on serious inference workloads - consistently outperforms attempts at going fully serverless with a large model. Before committing to any LLM deployment strategy, it's worth pressure-testing the cold start behaviour, the context storage plan, and the cost model at realistic request volumes. For more on the NLP skills that underpin effective language model pipelines, check out the Development episode on Custom Tokenization Pipelines: The NLP Skill You Can't Afford to Skip. DEV

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. The U.S. Has More Data Centers Than the Next 14 Countries Combined

    Nov 20, 20265 minAudio

    The global data center map looks lopsided by any measure - but the full picture is more striking than a simple site count suggests. This episode of DEV.co digs into a newly published graphic analysis showing just how...

    Credit the people on this episode

  2. From $30 Million to $5 Billion: The AI App Explosion by the Numbers

    Nov 18, 20265 minAudio

    The generative AI app market has posted one of the most dramatic growth curves in consumer software history - and the numbers behind that curve reveal a story far more nuanced than the headline figure suggests. This...

    Credit the people on this episode

  3. AI Adoption: Why the Numbers Are So Far Apart

    Nov 15, 20265 minAudio

    A single statistic can make AI feel like a done deal - nearly every organization already on board, the window to act almost closed. But a competing figure from the U.S. Census Bureau tells a starkly different story....

    Credit the people on this episode

  4. 83% of Developers Use AI Tools — But Only 6.6% Actually Trust Them

    Nov 13, 20265 minAudio

    Widespread AI adoption among software developers is no longer a trend to watch - it's the current reality. But a sweeping Stack Overflow survey of nearly 31,000 developers reveals a striking contradiction at the heart...

    Credit the people on this episode

  5. Private RAG Over Microsoft 365: Permissions, Delta Queries, and What Actually Ships

    Nov 11, 20265 minAudio

    Enterprise teams keep hitting the same wall when grounding LLMs on SharePoint and OneDrive: a working prototype that fails the moment security asks who actually owns access to the returned content. This episode of...

    Credit the people on this episode