Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Unsupervised Learning (Member Edition)

Unsupervised Learning (Member Edition) artwork

Using the Smartest AI to Rate Other AI

Apr 19, 202510 minAudio
ShareXLinkedInFacebookWhatsAppEmail

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs - by scoring them across a range of tasks and comparing them to human intelligence levels. I talk about: 1. Using One AI to Evaluate Another The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review. 2. A Human-Centric Grading System Models are scored on a human scale - from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower - just as expected. 3. Custom Prompts That Push for Deeper Evaluation The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance. Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest - but the framework and methodology still work perfectly with current models. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler See you in the next one! Become a Member: https://danielmiessler.com/upgrade

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Evolution, Lit RPG and AI Harnesses

    Sep 17, 202624 minAudio

    I went on a walk and ended up connecting three things I've been thinking about: evolution, LitRPG, and personal AI harnesses. Why listening to The Primal Hunter keeps getting me excited to work on my own goals, and how...

    Credit the people on this episode

  2. The Missing Piece Is Ideal State

    Aug 31, 202621 minAudio

    📝 Theo put out a video saying you should turn off Claude Code's memory, and it triggered me enough to record a response. My argument: the real problem with all AI is the gap between what's in your head and what the AI...

    Credit the people on this episode

  3. The Different Games OpenAI and Anthropic Are Playing

    Aug 14, 20264 minAudio

    Anthropic is going after the enterprise operations layer; OpenAI wants to be everyone's personal agent. Both are multi-trillion-dollar markets, and they barely overlap. Become a Member: https://danielmiessler.com/upgrade

    Credit the people on this episode

  4. Debating the Morality of Dario Amodei

    Jul 9, 20261 hr 15 minAudio

    A debate with Zack Korman on the morality of Dario Amodei specifically, and Anthropic in general. Zack's version of the video: https://youtu.be/mQ9-YoE9ykE Become a Member: https://danielmiessler.com/upgrade

    Credit the people on this episode

  5. Rethinking AI Harnesses

    Jul 9, 202612 minAudio

    Why we should think of the AI harness not as a coding tool, but as an Intent Engineering system that manages your entire life - a LifeOS. https://github.com/danielmiessler/LifeOS Become a Member:...

    Credit the people on this episode