Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Learning Bayesian Statistics

Learning Bayesian Statistics artwork

How AI Can Evaluate Bayesian Workflows with Bayesify

Sep 28, 20265 minAudio

Today's clip is from Episode 165, featuring Alex Fengler. In this conversation, Alex introduces Bayesify, a tool that uses AI to analyze research papers and assess how well they follow a Bayesian workflow. He explains how the tool breaks an analysis down step by step, identifies strengths and weaknesses, and provides suggestions for improving the paper. They also discuss how BasiFi can be used as an educational resource, a review engine for researchers, and potentially as a way to study how the quality of Bayesian analyses has changed over time. Alex also explains why the team is building a human-rated "gold set" of papers to evaluate how well the tool's scoring aligns with expert judgment. It's an interesting example of how AI can be used not just to generate research, but to help verify and evaluate statistical workflows. Full discussion here Support & Resources → Support the show on Patreon → Bayesian Modeling Course (first 2 lessons free): Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work!

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

speaker-0: We are working on this little tool that ⁓ we call it Bayesify. ⁓ it's a tool where you can go and upload a research paper, also your own, obviously, and get a structured report on strength and weaknesses stepwise on how you applied the ⁓ Bayesian workflow, right? W including then hints at how to how to improve how to improve the paper. Right. If it's your own, this can serve as a review engine. ⁓ to drive you to a good point. If it's ⁓ other historical papers, it serves as like a educational resource and later downstream possibly as a as a well of ⁓ data through which we can do some historical forensic analysis also on let's say the quality right of ⁓ Bayesian analysis over time. ⁓ so I could go here, right, load a paper. It will take like ⁓ a minute or two to to run this.

speaker-1: you

speaker-0: So I can just upload the paper here, right? But I also have it ⁓ preloaded on this on this tab. So this is like my own paper now, right? ⁓ Like Yude Approximation Networks for Fast Inference of Simulation Models and Cognitive Neuroscience, right? So what we'll get is this little report here. So what is the report about, right? ⁓ first it has a relevance gate, whether the the the paper is even you know in the realm of what makes sense to analyze through Bayesian workflows. Then we have a rubric here. So we have ⁓ we allow anchoring the analysis on very specific workflow papers. There's for example a famous ⁓ paper from Gellmann at all, a Bayesian workflow, right? And we have a few options and then we also have our we call it now the gold standard ⁓ synthesis, which is a synthesis of ⁓ a few such candidate papers, right, that is then projected into ⁓ a synthesis version of the stepwise approach and we'll always link to ⁓ any particular step to the sources that were used to define the step. Right. We we'll see in one second. And it ⁓ it classifies the paper type. So in my case it's method development. So this is not just an empirical data analysis. So it will then check which steps of the workflow are useful. Nine remain of an original eleven and it will give you a score on how many of them sorry, in this case it's ⁓ seven out of nine. Sorry. ⁓ and it will give you an overarching Bayesify score, right? So far it's been harsh on any of us who tried it. ⁓ have not seen a very high score. ⁓ did you try it on the actual Bayesian workflow paper by Gellman? It's a well calibration will be defined. That paper will be defined as method development. So we should you're right, we should do we should do that. ⁓ but since it's defining the framework, you couldn't necessarily ask it to follow itself the framework. That's ⁓ that's a recursive approach. but yeah, we are we are still working in generally speaking. ⁓ the goal here is to have ⁓ a what we'll call a gold set of like human rated papers and ⁓ test the engine for how well it's calibrated to real expert judgment of ⁓ papers. So we're currently in the process of collecting such a gold set so that we can check the scoring mechanism here against what experts in the field would have actually said. So you can get you'll get yourself this ⁓ report, right? Many s many different steps here, how well they are applied, ⁓ whether or not you were adequate. You can see I was adequate only on a few of them. ⁓ I have a couple of them missing in my paper. ⁓ in hindsight, lucky that I got it published. And then you'll You'll get suggested fixes. ⁓ it gives you the sources of evidence that were used to judge the step, right? And it will give you the the standards that were let me pull this in. The standards that were applied. Sorry, where is it? Here. ⁓ so this particular step, right, posterior predictive, ⁓ retrodictive checks, ⁓ is based around the scoring mechanism is based around these two papers the visualization invasion workflow ⁓ and the Gelman Bayesian workflow paper. So that's a tool we're we're working on. ⁓ it's very close to it's actually already reachable now. We have Basify.org. ⁓ but we're still making refinements to this to make sure that ⁓ eventually we can ⁓ properly claim that it's ⁓ solid and it also tracks ⁓ in a calibration sense real expert judgments.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. #169 Bayesian Workflow for Beginners, Alex at Princeton

    Nov 18, 202655 minAudio

    Support & Resources → Support the show on Patreon → Bayesian Modeling Course (first 2 lessons free) Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work!...

    Credit the people on this episode

  2. Why Start with Bayesian Multilevel Models for Repeated Measures?

    Nov 17, 20265 minAudio

    Today's clip is from Episode 168 of Learning Bayesian Statistics, featuring Santiago Barreda, who joins host Alex Andorra to discuss Bayesian multilevel models and why they are particularly useful for researchers...

    Credit the people on this episode

  3. Why Speech Researchers Should Give Bayesian Statistics a Shot

    Nov 12, 20264 minAudio

    Today's clip is from Episode 168 of Learning Bayesian Statistics, featuring Santiago Barreda, who joins host Alex Andorra to discuss Bayesian multilevel models and their applications in speech perception and...

    Credit the people on this episode

  4. #168 Speech Perception & Multilevel Models, with Santiago Barreda

    Nov 4, 20261 hr 26 minAudio

    Support & Resources → Support the show on Patreon → Bayesian Modeling Course (first 2 lessons free) Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work!...

    Credit the people on this episode

  5. How I built Learn Bayes Stats to the Top 1.5%

    Oct 28, 202623 minAudio

    This one is a bonus episode, where I was the guest, on a podcast called One Thing That Worked! Get all the insights and takeaways here. In the meantime, here is how they described it: In this insightful episode of One...

    Credit the people on this episode