Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Delta: HealthTech Innovators

Delta: HealthTech Innovators artwork

AI in Medicine is BROKEN: Stanford PhD Exposes the 95% Accuracy Lie | LLMs in Healthcare

Oct 6, 202536 minAudio
  • Apple Podcasts
ShareXLinkedInFacebookWhatsAppEmail

Is AI really ready to replace doctors? Stanford PhD researcher Suana reveals shocking truths about medical AI that Big Tech doesn't want you to know. When she tested leading AI models like GPT-4, Claude, and DeepSeek on modified medical questions, their accuracy plummeted by up to 40%!In this eye-opening conversation, we dive deep into: ❌ Why 95%+ accuracy on medical exams means nothing in real clinical practice ❌ How AI models fail when there's "no right answer" (which happens constantly in medicine) ❌ The dangerous gap between flashy headlines and clinical reality ✅ How doctors can safely use AI as a co-pilot (not replacement) ✅ The future of medical AI evaluation and what needs to changeSuana is a 3rd-year PhD student at Stanford in Biomedical Data Science, pioneering real-world evaluation methods for medical AI. Her research on MedELM and benchmarking is reshaping how we think about AI deployment in healthcare.🔬 Key Research Discussed: JAMA Open publication on AI robustness in medical diagnosis MedELM: 35-dataset benchmark suite for real clinical tasks Why MedQA and USMLE-style tests don't reflect actual patient care ⚠️ CRITICAL TAKEAWAY: AI models are trained to always give an answer, even when "none of the above" is correct - a potentially dangerous flaw in medical decision-making.📚 Resources Mentioned: MedELM Leaderboard (public repository available) Research on medical AI evaluation standards Real-world hospital deployment considerations Timestamps: 0:00 - Introduction: Why Medical AI Evaluation is Broken 1:04 - Suana's Journey: From Computer Science to Healthcare AI 2:32 - The 3 Critical Problems with Current AI Benchmarks 8:28 - The Research: Testing AI with "None of the Above" 17:24 - Shocking Results: AI Accuracy Drops 8-40% 19:02 - Why AI Can't Say "I Don't Know" 23:10 - Take-Home Message: Use AI as Co-Pilot, Not Replacement 24:58 - Real Clinical Examples: When AI Actually Helps 28:12 - MedELM: The Future of Medical AI Evaluation 34:35 - Final Advice for Doctors, Patients & Developers Whether you're a physician, healthcare worker, AI developer, or patient curious about medical AI, this conversation will change how you think about artificial intelligence in healthcare. Paper link: https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2837372

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. AI Won't Replace Doctors | Othman Laraki, Color Health

    Nov 16, 202645 minAudio

    Othman Laraki, co-founder and CEO of Color Health, on why he believes AI will increase the demand for doctors, and what it takes to run a virtual cancer clinic. In this episode of Delta: HealthTech Innovators, I sit...

    • Apple Podcasts

    Credit the people on this episode

  2. Cancer Care Happens Between Visits | Dr. Kingsley Ndoh, Hurone AI

    Oct 26, 202650 minAudio

    Dr. Kingsley Ndoh, physician and founder and CEO of Hurone AI, on building AI that supports cancer patients in the gaps between visits. In this episode of Delta: HealthTech Innovators, I sit down with Kingsley to talk...

    • Apple Podcasts

    Credit the people on this episode

  3. How OpenEvidence Became a $12 Billion AI Healthcare Giant | Full Breakdown

    Feb 24, 20263 hr 58 minAudio

    OpenEvidence just hit a $12 billion valuation - here's the full story of how it happened. Daniel Nadler sold his first company, Kensho, to S&P Global for $550 million. Then he turned his attention to one of the most...

    • Apple Podcasts

    Credit the people on this episode

  4. How Doximity Built a $9B Monopoly Doctors Actually Love | The Full Story

    Jan 21, 20264 hr 2 minAudio

    In this episode, we dive deep into the incredible story of Doximity - the "LinkedIn for Doctors" that captured 80% of US physicians and became one of healthcare's most under-the-radar success stories. Thank you for the...

    • Apple Podcasts

    Credit the people on this episode

  5. The Doctor Will Text You Now: How Councel Health is Scaling the "Clinician Cockpit" with AI

    Jan 20, 202635 minAudio

    Join us for an insightful conversation with Dr. Rishi Khakhkhar, Chief Medical Officer at Counsel Health, as we explore how this AI-enabled virtual care company is transforming the way patients access healthcare through...

    • Apple Podcasts

    Credit the people on this episode