Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DEV

DEV artwork

The Eval Gap: How to Know if Your Internal AI Tool Actually Works

Aug 26, 20268 minAudio

Shipping an internal AI tool is the easy part. Knowing whether it's still working six months later - that's where most teams go silent. This episode of Development tackles the "eval gap": the absence of any repeatable system to detect when an AI tool's output quality has silently drifted, and what it actually takes to close it. The episode walks through a concrete, step-by-step framework for building an evaluation practice around custom internal tools - from structured data extraction to open-ended response drafting. Key topics covered include: Why silent degradation happens - model provider updates, shifting inputs, and accidental prompt edits can all erode output quality without triggering any error or alert. Building a golden set - how to curate 20-30 representative historical examples with verified correct outputs, and why this investment pays off more than any other part of the process. Defining a rubric - the difference between binary scoring for structured tasks and dimension-based scoring for open-ended outputs, and how to make either one fast enough to actually run. Using a model as a critic - why prompting an AI to answer specific yes-or-no questions about another AI's output is a reliable evaluation method, and what makes it work. Setting thresholds with consequences - how to turn evaluation scores into operational decisions, so results on a reporting dashboard drive real action rather than sitting unread. The three failure modes - stale golden sets, evaluations too slow to run consistently, and waiting until something breaks before building the practice at all. The episode makes a strong case that evaluation isn't a one-time audit or a technical luxury - it's the operational layer that separates a tool that holds up over time from one that quietly becomes a liability. The earlier it's built into the workflow, the more useful it becomes as part of a broader business operating system. If you enjoyed this one, the episode How to Run a Debrief So It Actually Feeds Your Next Bid applies a similar operational lens to a different part of the business cycle. VB RFP

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. The U.S. Has More Data Centers Than the Next 14 Countries Combined

    Nov 20, 20265 minAudio

    The global data center map looks lopsided by any measure - but the full picture is more striking than a simple site count suggests. This episode of DEV.co digs into a newly published graphic analysis showing just how...

    Credit the people on this episode

  2. From $30 Million to $5 Billion: The AI App Explosion by the Numbers

    Nov 18, 20265 minAudio

    The generative AI app market has posted one of the most dramatic growth curves in consumer software history - and the numbers behind that curve reveal a story far more nuanced than the headline figure suggests. This...

    Credit the people on this episode

  3. AI Adoption: Why the Numbers Are So Far Apart

    Nov 15, 20265 minAudio

    A single statistic can make AI feel like a done deal - nearly every organization already on board, the window to act almost closed. But a competing figure from the U.S. Census Bureau tells a starkly different story....

    Credit the people on this episode

  4. 83% of Developers Use AI Tools — But Only 6.6% Actually Trust Them

    Nov 13, 20265 minAudio

    Widespread AI adoption among software developers is no longer a trend to watch - it's the current reality. But a sweeping Stack Overflow survey of nearly 31,000 developers reveals a striking contradiction at the heart...

    Credit the people on this episode

  5. Private RAG Over Microsoft 365: Permissions, Delta Queries, and What Actually Ships

    Nov 11, 20265 minAudio

    Enterprise teams keep hitting the same wall when grounding LLMs on SharePoint and OneDrive: a working prototype that fails the moment security asks who actually owns access to the returned content. This episode of...

    Credit the people on this episode