Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DEV

DEV artwork

Multi-GPU Training With Model Parallelism in DeepSpeed

Jul 9, 20269 minAudio

Modern AI models have grown far beyond what a single GPU can hold in memory - and that's not a problem you can optimize your way out of on one device. This episode of Development tackles the architecture, tooling, and practical considerations behind multi-GPU training, using Microsoft's DeepSpeed framework as the focal point. It's grounded in this in-depth guide to multi-GPU training with model parallelism, which is worth having open alongside your own training setup. The episode walks through the full picture - from why model scale has made distributed training a necessity, to the key parallelism strategies, to what a DeepSpeed implementation actually looks like in practice. Here's what's covered: Why single-GPU training hits a hard wall - at billions of parameters, even high-memory GPUs can't load the full model, making multi-GPU training a prerequisite, not an optimization. Data parallelism vs. model parallelism - data parallelism replicates the model across GPUs and splits the data; model parallelism splits the model itself, which is the only option when the model won't fit on one device. Pipeline parallelism and tensor parallelism - the two main flavors of model parallelism: dividing the model by sequential layer stages, versus sharding the matrix operations within individual layers across devices simultaneously. DeepSpeed's ZeRO Optimizer - rather than duplicating optimizer states on every GPU, ZeRO partitions them across devices, dramatically cutting per-GPU memory usage and enabling much larger model training runs. What a DeepSpeed integration looks like - the framework wraps around a standard PyTorch workflow; a JSON config file handles parallelism settings, and the core training loop requires minimal changes. Common pitfalls and practical guidance - the episode flags key traps including ignoring communication overhead, failing to re-tune batch size and learning rate after scaling up, and trying to combine every parallelism strategy at once before profiling incrementally. The real-world use cases discussed range from large language models and BERT-family architectures to massive recommender systems with embedding tables that routinely exceed single-GPU memory. The throughline is consistent: DeepSpeed doesn't eliminate the complexity of distributed training, but it makes that complexity configurable rather than something every team has to re-engineer from scratch. If you've been thinking about LLM inference infrastructure more broadly, the episode Why Your LLM Service Needs an Async Prompt Queue covers a complementary piece of the production puzzle. DEV

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. The U.S. Has More Data Centers Than the Next 14 Countries Combined

    Nov 20, 20265 minAudio

    The global data center map looks lopsided by any measure - but the full picture is more striking than a simple site count suggests. This episode of DEV.co digs into a newly published graphic analysis showing just how...

    Credit the people on this episode

  2. From $30 Million to $5 Billion: The AI App Explosion by the Numbers

    Nov 18, 20265 minAudio

    The generative AI app market has posted one of the most dramatic growth curves in consumer software history - and the numbers behind that curve reveal a story far more nuanced than the headline figure suggests. This...

    Credit the people on this episode

  3. AI Adoption: Why the Numbers Are So Far Apart

    Nov 15, 20265 minAudio

    A single statistic can make AI feel like a done deal - nearly every organization already on board, the window to act almost closed. But a competing figure from the U.S. Census Bureau tells a starkly different story....

    Credit the people on this episode

  4. 83% of Developers Use AI Tools — But Only 6.6% Actually Trust Them

    Nov 13, 20265 minAudio

    Widespread AI adoption among software developers is no longer a trend to watch - it's the current reality. But a sweeping Stack Overflow survey of nearly 31,000 developers reveals a striking contradiction at the heart...

    Credit the people on this episode

  5. Private RAG Over Microsoft 365: Permissions, Delta Queries, and What Actually Ships

    Nov 11, 20265 minAudio

    Enterprise teams keep hitting the same wall when grounding LLMs on SharePoint and OneDrive: a working prototype that fails the moment security asks who actually owns access to the returned content. This episode of...

    Credit the people on this episode