speaker-0: Let me show you how to be a good bee.
speaker-1: This episode is a special one. It was recorded live on stage at StanCon 2026 in Uppsala, Sweden, in front of a room full of STAN users and developers. My guests are two binars of the Stan world. Paul Birkner, professor of computational statistics at TU Dortmund and creator of BRMS, is back on the show after episode 35, and Steve Bronter, software engineer at the Flatiron Institute. And stand card developer joins us for the first time. With Paul, we dig into amortized patient inference, why some models don't have a usable likelihood, and why simulation-based calibration isn't enough to trust a normal posterior. With Steve, we talk about what's holding GPU computing back in stand, the push to move the whole algorithm onto the GPU, and his work with the eigen team on GPU-ready linear algebra. Along the way, we cover some to zero constraints. Leave one group out cross-validation, a very clear definition of the Jacobian adjustment, how both of them use AI tools, and some great questions from the audience. A big thank you again to the StanCon organizers, especially Sarah Hammis, Mans Magnussen, and Aki Vetari for inviting the podcast to StanCon. This is Learning Basion Statistics, episode 167, recorded August 20, 2026.
speaker-0: So you had to be a good beesie And change your predictions after taking information And if you're thinking I'll be less than amazing Let's adjust those expectations What's a Bayesian? It's someone who cares about evidence
speaker-1: Welcome to Learning Bijan Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andora. You can follow me on Twitter at Alex underscore Endora, like the country, for any info about the show. LearnBaseStats.com is Lab Place2B. Show notes, becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon, everything is in there. That's LearnBase Stats. dot com. If you're interested in one on one mentorship, online courses or statistical consulting, feel free to reach out and book a call at topmate.io slash alex underscore and Dora. See you around, folks, and best Paigian wishes to you all. Hello my dear Bayesians. Just a short announcement before continuing with the episode that I will be at the Carnegie Melon Spots and Analytics Conference in Pittsburgh, Pennsylvania on October 23rd and 24th, 2026. And on October 23rd, I will actually teach a workshop about Bayesian inference. You guessed it. And also causal inference and how to use Bayesian and causal inferences. To infer the effect of a player on his team. So if you wanna come meet me and also attend my workshop, that would be awesome to see you there. And the idea will be to teach you how to causally infer the effect of a player and the effect of a team and how to disentangle them with one model, which will be the soccer factor model that I'm working on with Max Gabbel. It will be actually the updated version of it. So if you're curious about causal inference, Caution processes, time series, and PyMC, I am sure there will be something for you to learn there. You don't have to be interested in sports analytics or football or soccer to get something from it. If you are, well that's great because that's a sports analytics conference. But if you want to apply that in your marketing models or biology models or whatever, you will definitely get something from that because what counts is really the concepts and The time series data and the structure that we will be looking at. I put all the details in the show notes. You will have the link to actually sign up for the conference. So hopefully, see you there on Friday, October 23rd. And also a big thank you to New York Co for inviting me in the podcast over there because actually my workshop will also double as the Learn-Base Stats episode. You can find more about Ron and his background and his research actually in sports analytics in episode one forty of the podcast. That was a very fun one, so I definitely recommend you tune in for that one. And on that note, let's go on with the show. welcome back. Now a special live recording of Learning Bayesian Statistics podcast. it's my favorite podcast. and Alex Andorra is the creator of the post podcast. I highly recommend it. And it's been great to hear thoughts from many basiens and it's I'm certain that it's been also helpful for making stan and TAN developers and different parts of TAN, like more widely known, spreading spreading the information. and then we have Paul Berkner, Steve Bronder as the guests for this podcast. But please, Alex, take over. Yeah, thank you so much, Aki, for the the great introduction. I'm very happy to hear it's your favourite podcast. it's definitely the only Bayesian podcast, so by definition it's the best and the worst at the same time. So that's that's a very rare qualification. if you don't know the podcast, that's very hurtful. But you have a QR code at the very top here that will bring you to the website and you'll see that we have now one hundred and sixty three episodes since twenty nineteen. Aki has been on there, Paul and now Steve, and so that's why the The podcast is is great. Definitely not thanks to me. so you'll see lots of episodes there. next week is gonna be episode one hundred sixty-four. That's gonna be a very special one. I recommend tuning in for that one. And the second QR code is something very new. I've started an Instagram account for the show to give a bit of the behind the scene feel because I get a lot of questions about that. So if you wanna see Charles Margaution kicking my ass at football, Paul Björkner being very good at football, actually and me being very bad, you can you can go in there and you'll see that later. but today we're gonna talk with Paul and Steve. We're gonna open to QA more or less ten minutes before the end of the recording. So if you have questions, write them down. And and we'll get to them at the end. But right now, let's focus on that. Feel free to, you know, like upload, make noise, things like that, because this will be real podcast episode. So if you wanna be on the show that's it's your time now to be a heckler or something like that, you know. Okay, so let's start. I recommend tuning in to episode thirty five. I think that's the one where you were on. Paul, so but that was a long time ago, so I'm very happy to have you back today. It's definitely awesome. but if you want to have like the full background and history of Mr. Buckner, that's gonna be in that one because today we're not gonna do that. however, Steve, it's your it's your first time. how are you feeling?
speaker-2: A little anxious to be recorded. I always feel little anxious. I'm kind of a background character, I feel like, in a lot of stand stuff. And so it's kind of like I don't usually show up on main stage.
speaker-1: Well, I'm I'm very happy that you're here today and that you're here even though you're a bit anxious. Thank you so much. because yeah, you're doing so so many amazing things that we just saw a bit, we had a bit of a window in there. But I always like to dive a bit more into, you know, who the people are on the show. So yeah, Steve, can you give us your origin story? Basically, what do you do today? Because I think you have a very original job, you know. It's just like what what do you do? I'm a stand core developer. What what's your job and how did you end up doing that?
speaker-2: Yeah, so I am a software engineer at the Flatiron Institute, working under Bob. a lot of my job is maintaining and helping building things out for Stan, as well as like when we have new contributors helping shepherd them into becoming more long-term developers, if someone has a feature making sure it comes in and it's up to the standards so that we can be able to maintain it in the long term. in terms of how I got to where I am, In undergrad, I actually started as a political science and journalism major. Re. Yeah. I like thought I would just like I just really like wanted to be like an on-the-beat style, like, yeah, like what's really happening in politics. and after a while I just kind of like I had a bunch of this sounds so silly. I had a bunch of friends who were like electrical, or they were like engineers or like electricians. And they made like $50 an hour. And I was like, look, that's at that time I was like, like that's sick. I kind of want to I kind of want that. And so I was like, you like I don't know if I like school or not. I am going to pick a major where like I have a very high likelihood of dropping out. like I'm gonna actually try, but like I am gonna be open to failing and like that's okay. So I did econ and math. and I ended up really liking them. So I actually completed my degree and everything and then just fell in love with like forecasting and stuff like that. I then went from undergrad to Columbia to do a masters in their quantitative methods in the social sciences program that andrew gelman help run. and at that time I took a class on GPU computing and deep learning because I was very pretentious at the time and I said, well I'm gonna take these classes just so I can talk about them at parties and tell people how stupid they are. And I did these classes like okay GPUs are cool. But like I was like, you would need so much compute to make deep learning actually work. I just like don't think this is gonna be a real thing. No one's gonna ever do this. And that was the most wrong I've ever been in my entire life. Like I readily do admit that.
speaker-1: Yeah.
speaker-2: Yeah, yeah, you're right. Like I I have to make up for my mistakes. but I was doing this GPU stuff and Ben Goodrich was my master's thesis advisor on a very non-Bayesian thing, bless him. and he was Well, we want some GPU stuff in Stan. Would you be willing to help us try to work on this? And I said, totally. and I wrote like the first implementation of Stan's GPU code. And like as we were about to release it, Rock Chesnevar showed up and was like, hey, like we're like also working on a GPU for Stan thing. Like what do you think about our code? And I was like, like this is so much more efficient than my code. but it was like written a little bit, I was like, you know, like let's work on making this let's integrate all of my stuff together. so that's kind of how I started working on Stan. and then I went into finance and I was doing structured product stuff for a while and it was fun. And just kind of working on Stan like in my at home in my underwear at night. and then For after a while I was kind of complaining to stuff about Bob and he was like, like, well, why don't you come up to Colombia? we can like you know have you come and work for a while. And then Aki Aki also let me take a sabbatical in Finland, which is like one of the best times I've had in my life, and I'm very thankful. Love Helsinki. and so I kind of got back into being like a main contributor for Stanigan, and then kind of moved to Flatiron from Columbia, kind of f following Bob. And so that's kind of how I got to where I am today. This is a long winded way to say it, apologies.
speaker-1: No, that's that's perfect. That was great. Thanks. yeah, I didn't know you had started that way, so you you really like that was a very senior senior way. So that's great. I mean the I'll what I really like about these paths usually is that you end up doing something because you really want to do it and really like it, which I think is definitely your case. what's interesting though is that I find your Profile is much more geared towards, well as you were saying, software engineering and really kind of like, you know, the the background of offstand and what we're using and how we're actually getting the the results. more than on the statistical and modeling side and actually getting into that because you wanted to do Bayesian statistics. Is that is that yeah, am I correct here or feel free to say I'm completely wrong.
speaker-2: I think you're right on a lot of points. Like I think that my undergrad was very focused on stat stuff. But then over time I kind of just ended up liking software engineering and like the computational side more. and so that kind of ended up being my focus, yeah. I still love stat stuff. I do like a bunch of weird little like stats. Most of my day-to-day like weird C stuff. but like I still have like weird side projects that I just are like being able to riff and like fun little Bayesian models.
speaker-1: Yeah, yeah. Okay. I think later we can come back on that you know GPU stuff that you that you mentioned. I think that'd be interesting because that's also top of mind for a lot of people right now, obviously. but now Paul, let's let's talk a bit more about you. So I'm guessing most of the people you know you interact with in the Bayesian world, they identify you as the BRMS guy. Is that is that correct? Like do you get recognized sometimes in the hallways and you're like, Aren't you the Bureau Miss guy?
speaker-3: It would certainly be good for my ego, but I'm not sure this has happened so much.
speaker-1: Maybe because you didn't put the picture put a headshot of you on the README of Pure MS, you know, and then people start recognizing.
speaker-3: Here. could be, it could also be that I kind of moved out a little bit of out of the fields, like, I don't know, psychology where I'm originally from, for example, where BRMS is a lot used, people may or may not recognize me and moved more like to computer science, deep learning aspects, where nobody probably very few people know Stan in the first place. Only if they know to compare MCM, CSAM +, nobody knows BRMS or do they care about R or something. So it's like, it's always humbling to go to a new community. where you are kind of completely like a blank person, you have zero standing.
speaker-1: Yeah, that was the that was the segue actually of my question. So yeah, when you came on the show that was to talk about BRMS, very BRMS for cust. But you do a lot of things. And like so we've seen one of the presentations, you were one of the co authors. I know you do a lot of stuff on GPs, HSGPs, you love them also. and you'll obviously do a lot of things more on deep learning now, especially with Stefan Ralev and his group. so Yeah, if you're interested in that by the way, amortized patient inference, definitely recommend checking that out. the base flow package. we have a bunch of episodes about that on the on the website, so definitely listen to them. But yeah, like you you wanna talk about us, you wanna talk about that, maybe like what you're doing in that in that field right now.
speaker-3: Yeah, I can talk a little bit about the history how I got there. So when I was like working on BRMS and Stan and all sorts around that, things ended up working so well, you know, that I got bored. So I had to move away from Stan a little bit because it used to and still works so well. Still surprising to me. And so I met Stefan, one of my close collaborators now in a BRMS workshop before COVID pandemic, right before COVID pandemic. He didn't need to be in the workshop at all. I don't know why he went there, he do the psychology PhD. And then he showed me this paper on the original kind of, using essentially neural networks as a replacement for MCMC. So the statistical Bayesian models remain the same, but our estimation is now done with neural networks rather than with MCMC, which has advantages because we can train on simulation data so we don't need likelihood densities or prior densities. And we get real-time inference if we do it well, but of course we are having all the pain that neural networks come with, lack of guarantees, unstable training, et cetera, et cetera. He was showing me one of the papers which recently got rejected by WeirdReviewer2. So I was reading that, I was really excited, and then we were like, yeah, mean, there's some COVID pandemic happening, COVID something happening in China, or like in Northern Italy, will not really hit us, then two weeks afterwards, everything will close. Also great at predictions.
speaker-1: Hehehehe
speaker-3: And then from there we worked on a lot. I was just bringing actually the expectation that I want these kind of amortized Bayesian inference methods to make them as reliable and robust as MCMC. I have no idea whether we ever achieve this, but it's like a goal that we can work towards. But multiple orders of magnitude faster. in the sense that after like new network training, which may of course be slow, then you get a new data set, you have the posterior distribution, and then you get another data set, you have the posterior distribution. So that's really impressive. So the promise is huge. And the challenges are also used, so that's why I was excited to work on that. But I also know that Stefan and I, of course, have been talking a lot to people, like also who are like, stand adjacent or even like part of the stand team, of course. And you know, for a lot of things like where we have all the likelihood densities available, like everything is kind of coding up and standing, we don't have like, I don't know, tens or hundreds of data sets that we need to run the same model on. We don't need. We are just happy with Stan or other probabilistic programming language, so there's a certain community that I'm also building methods for, and I'm not sure if they're ever going to be applied or appreciated by the community, but we will see.
speaker-1: Yeah, I mean and right now so basically what Paul said is is my just the let's say definition of amortized patient inference and why that would be even helpful. Have you now that you've been working on that for a few years now, have you seen any any interesting applications of these kind of methods, especially using base flow of course?
speaker-3: Yeah, we do have a lot, especially when we have models where we don't have the likelihood densities available or easily available and we just, but we can efficiently simulate from these models. So we then assimilate based Bayesian inference, which we did have like approximate Bayesian computation for decades, but it didn't really scale beyond, I don't know, a few parameters. And neural networks really humiliated when they came and humiliated all the benchmarks and made SPI really feasible and scalable. And as a side project, we got this amortized property that gives us real time basing. And so all the kind of models like physics models or sometimes models from psychology or biology or chemistry where we don't have these analytic likelihood densities which you don't see at Stencon of course. For these the kind of simulation based number test methods are like really like the state of the art, if you will. But of course, they would still not reach the reliability specifically for out of distribution data for which the model is misspecified, then MCMC would have in comparison if we could apply MCMC. So a lot of the core applications where this has been really successful is in places where we cannot efficiently do MCMC in the first place. Because if we did, probably MCMC would still win at least in the accuracy and the the trustworthiness that we get out of it. It's my goal to improve that trustworthiness of the neural methods so that we get faster inference but similarly accurate.
speaker-1: Yeah. Can you
speaker-2: I gonna is there is there methods yet or measures that you can use to say, well this didn't quite work as we wanted it to?
speaker-3: So in a way, So basically, there's, I mean, of course, know simulation-based calibration, right? And for MCMC, it's kind of, or variance of this is kind of gold standard in a way. But it only works, and you know that I'm just giving for context, it only works, of course, if we are simulating from the exact same model that then we are essentially fitting, But of course if that works, let's say with MCMC implemented in standard or wherever, then we have reasonable trust that also for real world data which may look a little bit different than the simulated data, still we get an accurate posterior approximation. So now for kind of neural simulation based armatized inference, of course we can run SBC, but it's more of a convergence diagnosis because we have been training the neural networks with exactly this kind of simulated data. And we also know that on the neural network works are only good to predict something that is close enough to the data that seen during training. So while for SBC we may easily pass, for example, we don't know if it will work for real-world data. if the real world data is somehow out of distribution of the simulated data. Because our models are usually misspecified in some way or form. So that there's, and this is actually what we're working with, to trying to diagnose not when the data, when the model is misspecified for a given data, but when it is misspecified in ways that will affect our... accuracy of the analytic posterior, which of course we don't know. So it's kind of a black box and the way we're doing it kind of, we are having two new networks. One is actually the posterior estimation and the other one is summarizing the data into ideally, like informative, ideally sufficient statistics. So we can check whether we are out of distribution in the space of sufficient statistics and if we are, chances are our posterior approximation will also be bad, which kind of gives us diagnostics and we have a paper soon coming out of this that extends previous methods. of hours and sometimes it works and sometimes it doesn't. So I'm still not happy.
speaker-2: We with embedded the plus we also had similar issues as well.
speaker-3: Yes.
speaker-1: Thanks Steve for the the assist. I appreciate that. And yeah, actually to make sure people understand if that could be helpful in their case, you know, SBI, so simulation based inference and ammon amortized patient inference. can you explain because you said they were especially useful when there is no likelihood. Right. So Why would that even happen? You know, like can you explain to people what that means to not have a likelihood available?
speaker-3: Yeah, so for example, think your model is expressed via like a differential equation. I don't know, some epistemological models about, you know, we can use COVID, right, about the changes in cases and the people who are like infected and immune, et cetera. And of course, if the models are simple enough, these differential equations either have like an analytical-ish solution or we can use kind of some... kind of differentiable ODE servers as we have in Stan. It will slow down things, but it will kind of work. But as our differential equations expressing our mechanistic model, let's say in physical chemistry or biology, gets increasingly complicated, we may not have... solve us in the way that we can solve through them and compute the densities and specifically the gradients in any numerically stable fast enough manner. So it's not that we would have no way of ever computing the likelihood density or prior density for that matter. It's just that it becomes so computationally expensive and numerically unstable that de facto it's like we say it's intractable in a way so that the numerical approximations don't work, but simulations from these models. easily doable differential equations, like partial, whatever differential equations, one aspect, the other one, you have like big latent variable models, like latent variables in psychological sense, you have, I don't know, theta your parameters, then they go to Z1, then they go to Z2, they go to Z3 and Z4 and Z5, and then you have Y in the end. And you don't care about Z1 to Z6, they are huge, there are thousands of latent variables, you don't want to estimate all of them. So you would like to use the marginal likelihood, marginalised all of these latent variables and you have just no way of computing this. So then it becomes de facto intractable because there's so much stuff in between that you just want to, you know, not having to deal with. And if you simulate, you simulate your theta, you simulate all your latent variables, you simulate your y, you throw the latent variables away, never look at them again, and then you have the relations of theta and y in simulations, that's all you need.
speaker-1: Yeah, yeah, thanks. I think that was very clear, but we we'll see if we have more questions about that. Feel free to ask later. so now Coming back to you to you Steve, I heard that neural networks they are quite expensive to to feed, right? So often we need a GPU for that. So you made fun of yourself earlier for making that forecast about GPUs, deep learning, it would never work. I'm not sure even though it turned out to be a wrong forecast, I'm not sure it was a bad forecast, probabilistically. Right, maybe at the time when you were making that forget that was actually the most the most reasonable one. So I guess now your like your vision of that has changed. Can you can you tell us what's your yeah, how you're thinking about GPU computing these days and and yeah, where where do you see that coming, especially when it comes to STAN?
speaker-2: yes. So my my forecast was in twenty sixteen. So was like I think if I had done a little more homework, maybe I could have made it more informed, but anyway. how I think about GPU computing is mainly that like as I kind of said in like the talk that we had before this, the main thing that stops Stan's current implementation from being as good as we want it to be is that it really just resides inside of the function that we use for calculating the log likelihood where what you really want is something where the entire algorithm is moved over to the GPU. Because for the RAM to communicate with the CPU it is pretty fast. but for the RAM to then communicate with the GPU it is just very, very slow. and so you want to because that's why you kind of want the whole algorithm over there. right now the kind of way I'm thinking about it for at least with Stan is that there are some ways that we can do that and port a lot of our back-end stuff to the GPU that could take a lot of time. Andre I was just speaking with had some schemes and ideas that I think are great. In terms of overall GPU computing stuff, I'm actually working with the Eigen team on a nice sort of C linear algebra library that would the the end goal for people who don't know, Eigen's a C library for linear algebra that essentially like if you use scientific software, it is almost surely that does linear algebra, it is almost surely using Eigen. It has a very pretty API front end and me and another guy are trying to figure out how we can get this front end to also work for GPUs. so that's kind of what I'd like is to have like a lower level GPU access schema. 'cause things like PyTorch and stuff you can totally just go do. GPU stuff right now. For some particular applications, like for autodiff, we do have to be really careful with memory management. And that means like we need to be able to actually go in and say, like, I need memory for this now, it must stay over here. and things like PyTorch don't quite give us like the accessibility that we need. and so right now that's kind of how I think about it. there's also stuff like Jax, where like I I use Jax and I like it. it means that like you can't write for loops, for instance, which is Like I, as I said before, like I like for loops, I think they're cool. but like they're incredibly hard to optimize over. so for Stan, we're working with another contributor. Me and me and Brian are working with another contributor to allow for people to write for loops, and then in some cases, we can collapse them such that they could then be ported over to the GPU. Jax also has these Vmap style functions, and we're also trying to port those into Stan because the Then once we see those, we can better reason about can this go over to the GPU or not. And so it is really a lot of backend machinations to allow users to write code just in a very natural way that can still be ported over to GPUs. Does that answer your question?
speaker-1: I yeah. Yeah. and is that something you're gonna focus on especially in the coming month or what or what's another thing that's gonna take most of your time, you think, in the in the coming month?
speaker-2: right now this sort of eigen GPU library has extremely beta. it's like super we're very in like the rough edges of making things how we want, figuring out what the API should look like. in the coming months it's definitely gonna be a big focus for me, just because like I think it'd be nice to have for a bunch of other like we're building it so that Stan and a few other libraries we're like very specifically targeting can go use it. And so that is like my kind of main focus and then doing it such that then we can essentially reuse a bunch of our current code, but then over on the GPU.
speaker-1: okay, yeah. Sounds like a yeah, hard problem to solve.
speaker-2: Is yes, it's there's a lot of not nice words to say for it, but it's gonna be a big effort for sure. Yeah. but it's it's fun and I like doing it.
speaker-1: Yeah, yeah, no, for sure. I mean of course. And something I liked so when you just did the the presentation earlier with with Brian about yeah, like basically update on where Stan is, I took some notes because there were some interesting things in there for me, like the topics I that are dear to my heart, especially of course the sum to zero vector and matrix. So I worked on that with Adrian Zable, who's just over there who spareheaded that effort on the on the Prime C side. If you're curious about that folks, episode seventy-four with Adrian. That's the first one. He explains what you what a sum to zero vector even is and why you would even want to do that. And it turns out, as you were saying, you definitely want to do that a lot of the time. It's gonna make your models much more efficient. And then we had Sean Pinkney. I don't see you here, Sean. But Yeah, I think it was episode one hundred thirty three. So if you wanna hear how that was developed on the stand side, definitely give that a listen. But yeah, I'm curious, so you talked very briefly about that in your presentation and you were saying that you were yeah, you were saying definitely a a big advantage of using that on a on a bunch of models. yeah, do you do you wanna add anything on that?
speaker-2: I think Sean did an awesome job of kind of explaining it. he I I love Sean. Every once in a while we'll just talk on the phone and he'll be like, yeah, like by the way, like here's this like I'll mention some like weird problem to him and then like a year later like, by the way, like I like totally figured out how to do this and I'm like, that's insane. and he's just a really fun guy. and so I think he probably said a lot of it more than I could or I think Paul would probably be able to talk about no, you give me the look of this is this math is also beyond me. Yeah, I agree. I agree.
speaker-3: Whenever I think about it, the basic parts are clear and then I want to implement it in BRMS and then Adrian made me aware again of some of the edge cases when things get like we get multiple grouping factors and they interact and then things get really really messy and then suddenly I'm scared of implementing it. So there I hope that someone whose math I trust will tell me if whatever and someone ends up doing in BRMS or whatever is correct.
speaker-1: Mm-hmm.
speaker-2: There's still a lot of talk with like I think there's been some discussions right at StanCon about like well like what if you have like correlations between the like parameters inside of the sum to zero vector? Like how do you 'cause it's just as simple as like, okay, like multiply it by a Cholesky correlation Cholesky of a correlation matrix, or like do you want to do some other schema to be able to set priors? There's still I think stuff we have to figure out before we can like how do you say standardize it in something like BRMS? but like I think it's very doable. It's it's foreseeable.
speaker-1: Yeah, yeah, yeah, yeah. so just to set some context here for people who are not familiar with sum two zero, it basically each time you're using a categorical predictor, for instance, in in a regression. The classic way of dealing with that and have your model not over-parametrized is to do I think it's called reference encoding, right? And so you take like you consider one of the categories as as your placebo if you want, and then everything is in comparison to that reference. But often turns out there is no reference category, and so the sum to zero handles that over-parameterization by fixing the mean, right, Adrian, or the sum. Yeah, that's the I mean I think it it turns out to be the same. So it's very intimidating to explain that when Adrian is in the room, by the way. so yeah, but anyways, that allows you to do to not have your model being over parameterized without having to fix a reference category. So for instance, super helpful if like I used to do a lot of electoral forecasting. you don't have a reference party, in France for instance. So then you just look at the at the mean of the parties. And so that's great, but then yeah, as you were saying, when you start having a lot of categorical predictors in your models and I don't know, you have a multinomial likelihood, then it starts being it starts getting messy. But it's extremely powerful and you you you are able to do a lot of of very interesting stuff with that. So I definitely encourage you folks to Use that in your models more often because more often than not, it will very probably sample much faster. I think I had another point about that, but I forgot. yeah, actually a question I had when I saw your slides earlier, which is a term we hear all the time in the Bayesian world, but I think is almost never defined. So maybe one of you two can define it. It's Jacobian. It's like all the time people talk about Jacobian. I think I think it's important we actually define okay what what's that and why is it important and why do we talk about that so much? Paul you wanna take a
speaker-3: I try to explain it to my students. Usually in school they had to do integration, and I hated integration in school. So nowadays I have two ways to do integration. is from whatever, Mathematica or LLMs, or Ravier, Ravier loves integration. So whenever there's a difficult integral, I'll it to Ravier. Ravier will solve it and be happy about it. And I'm happy because I don't have to do it. But anyway, so then probably people learn integration by substitution. when we substituted something, I don't know, we had x and then x was difficult, the integral was difficult to solve after x, then we just transformed into y, y is equal to f of x, but we cannot just plug in f of x and y, we cannot just change it, but we need to adjust the integral and we need to adjust the boundaries for the integral to remain valid. That was always very awkward for me to get right. And essentially, this adjustment terms that we not integrate over x but we're integrating over y and they are related in some kind of nonlinear manner. This adjustment term is kind of the Jacobian adjustment. Of course, in densities we need to make sure that they continue to integrate to one and if we just like plugged in f of x for y or the other way around, then the density would no longer integrate to one so it wouldn't be a proper density anymore. So essentially we have like just messed up our math and that's why we need the adjustment term which is Jacobian adjustment in multivariate terms. In multivariate cases it becomes kind of... the determinant of a matrix, it's kind of bit more messy.
speaker-1: Right. Beautiful. Yeah. Thanks, Paul. I think I'm gonna make a YouTube short so that part it's gonna go viral in the Bayesian world. But I think it's great. So Jacobian, another term you hear all the time is Kolevsky decomposition. Like for sure I'm just thinking about that right now. and also mass matrix adaptation. And that so episode one sixty three, thanks to Adrian Zabolt, actually I understood what mass matrix adaptation is and and why it's useful. So Give that to Lisa if that's something you've been curious about. yeah, Steve you wanna you wanna add something?
speaker-2: For mass matrix adaptation?
speaker-1: I don't know about anything we just mentioned now.
speaker-2: I worry that if I say something I'm not an expert in this place. I think if I say something, I'm probably gonna fudge it up a little bit. There's probably people who have been on the podcast who are gonna be much better at talking about.
speaker-1: But I think it's also interesting if you have questions about that actually. because I mean if you as a stand core developer still sees stuff like that that you're like you you can't really understand, I think it's that means very probably a lot of people have these questions. And we have a lot of people in the room that probably can answer that question. So like yeah, feel free to also ask those.
speaker-2: Mm-hmm.
speaker-1: Do you?
speaker-0: Yeah.
speaker-2: I'll think for a minute. Not right now.
speaker-1: Cool. something no before asking that question, I definitely wanna talk about what you're up to for the coming month, Paul. Because I know I know you do a lot of stuff, so yeah, I'm I'm always very curious to hear what you're what you're working on right now and what you're paying attention to.
speaker-3: So the hardest part at the moment is trying to find motivation to write more grant proposals.
speaker-1: Hehehe.
speaker-3: It's like I had grand proposals and then you know you can write follow-up ones and I should do it and I promise to do it and then it's hard and so I need to finish these. But research-wise it's kind of the diagnostics that I talked about how to diagnose whether out of distribution data will actually cause any issues for the posterior accuracy of ABI without knowing the ground truth and posterior not having MCMC reference. And also something we have observed is that sometimes when we have some of these statistical models and we learning these amortized posteriors with new networks. And let's say we have five parameters for the purpose of the argument and four parameters are just very well learned and the joint posterior is very well learned. There's one parameter where we just learn the prior. So we don't really learn how to condition on data properly. So we just recover the prior there. So we're learning something as opposed to nothing, but we're not really learning what we want. And of course, this could mean that the parameter in that statistical model is truly not identifiable, but usually we know it is. And if we run MCMC, if we could, then we would see a good recovery of the parameter. But we don't. And we figured out that there are some issues with the neural networks, the implementations, but generally how they stick together and how basically the gradient flows through them, which then leads some of the neural networks to learn to ignore data for certain kinds of parameters. So we have figured out some architectural changes and additional loss functions that we can employ to help the gradients move in the right direction so we can more reliably learn. join posterior of essentially all parameters and not ignore certain ones. That's something that will hopefully be done soon. I currently have four people doing four case studies. Let's see how that works, if this is going to speed things up or slow things down. I think that's one of some of the main things. And then something that I used to work on, which we have a paper coming out soon, is about leave one group out cross validation with Aki and Anna, both being in the room, and Svenja, one of my PhD students, who, and basically where we try to make this kind of. almost as efficient as loose CV with important sampling, but in a hierarchical model, if we're leaving out individual groups, and there's a bunch of approaches we can do, and I think we have found good trade-off between accuracy and speed there. So this is almost done, but just a little bit advertisement.
speaker-1: okay, yeah, super cool. So that'd be to make sure everybody understand, leave one group out. So that'd be logo. Yes. that's nice. Yeah. That's nice. Better than logo.
speaker-3: severe.
speaker-1: Cool acronym. So that means I guess it's for hierarchical models, right? So that'd be you have by definition groups in your hierarchical models. And what you're doing is that you're holding out one of the groups per se and then evaluating how the model does predicting that group out of sample.
speaker-3: Yes, exactly. Because if we had a hierarchical structure and we would do loose CV, then we would predict only single observation within an existing group. And that usually would do very well because we have all the observations of the same group still in the data set. So it often makes sense to just, if the task is to predict a new group, then to leave also the groups out. But there's just like a lot of difference between the full posterior and the local posterior without that group. So it's very hard for important sampling to bridge that gap, or usually impossible. So we have to do some smart integration. of parameters to make it work.
speaker-1: Yeah. Hm, yeah, okay. Yeah, because basically you can with hierarchical models you can predict a new individual in an existing group. So existing group here means like the model has seen in seen it during training, but you can also try and predict a new individual in a new group. So like completely new group that wasn't seen at at training. and that's obviously harder. and so you want here that be a way to understand before having to do that kind of prediction how the model you're working on is Like how good the model is at that task.
speaker-3: Yes, and in a way that ideally of course we can implement this in BRMS or something so it's just one click of a button and there's a little bit of more work involved to make it work like this but I hope you'll have algorithms that can be done.
speaker-1: Yeah, yeah, That's also where so that's a funny thing where sum to normal can be kind of a like a a solution that then brings a problem because if you do sum to zero sum to zero, sorry. if you do sum to zero on the existing groups but then you want to predict a new group, that means you're breaking the sum to zero, I think. looking at HR
speaker-3: depends on the level. if you correct within Stan for the sum to zero constraint, so what comes out of Stan is as if you didn't have the sum to zero constraint in the first place, which is kind of what we do, then we do the post-processing in R, Python or wherever, then we don't have to bother. But if we kind of just didn't do the correction in the middle, I don't know, things would become very messy and I would prefer not to think about it.
speaker-1: Yeah. Yeah. I've I've thought about that a lot because I use that a lot for hierarchical models. especially in spots analytics that happens a lot because you have new players coming in and but they are part of a group. And so yeah, like it's it's definitely it's definitely hard. The best solution I found is either you're lucky enough that you're over parameterized model, so without the sum to zero parameters You know it's over parameterized, but it still samples fine because you have enough data and so HMC is is fine. If you're not lucky enough to do that, then you need to use the sum to zero constraint, but then the out of sample predictions, what I would do is that I would kind of write a new model that I would but I would not sample from it. I would just do posterior predictive sampling on that model, but using the trace of the previous model. So it's like you have a new parameter, but that's informed by the previous run. It's bit artisanal, I think. I'm not sure it's the best mathematical solution, but glass does work. Yeah. yeah, I still have a lot of questions, of course, and I have the last two questions for you guys at the end, but we can already open the floor to the questions actually. So are there already some questions? Yes. So I think I have a mic over here. yes, perfect. Thank you, Sarah. Here. So you will be officially in the podcast, thank you so much. And I love reinforcement learning, so you will get a sticker at the end.
speaker-0: Thanks. my question's for Paul. in simulation based inference, like neural networks generally we don't have a very easy way of like gauging their certainty in their predictions for given parameters. So like is the posterior in simulation based inference does that widen or contract depending on how much training data you give your neural network?
speaker-3: So I'm just rephrasing the question, tell me whether I understand correctly. So of course, when we do simulation-based inference, way I do it, we still target what I call the analytic posterior that also MCMC would target, would we have MCMC available. Then of course, the question is, given infinite amount of training and infinite expressible architectures, we will converge to the analytic posterior. That doesn't help us practically, but theoretically, we will converge to the right thing. Then of course the question is what happens, let's say our architecture is flexible enough but we have, our training is not sufficient so we haven't converged yet. And then it depends on the loss function and the kind of architectures, whether the approximate posterior that is kind of... incompletely trained is more like over underestimating the uncertainty. So since we are usually training with forward KL loss, we can do it because in simulation with inference, we need to just simulate from the model, which is something we can do with a little bit of tricks. So we're using forward KL, so it's more like mass covering. So it will probably rather overestimate the uncertainty or in the cases I just earlier mentioned, just estimating the prior, which is kind of an overestimation, right? Often. But if we use different kind of loss function, let's say we would use for the purpose of the argument reverse KL loss, we would probably not be mask covering, but we would like like miss modes that we would be underestimating. So it depends a bit on the loss functions. What theoretically happened in practice, I have seen all kinds of things going really wrong, but mostly it's just that certain parts of analytic posterior is just not learned parameters and like like not learning from data at all. it's mostly that our posteriors are too wide, which could be the conservative place. But of course, if we were to just learn the prior or only learn from half of the data, that's too wide, but not really a satisfactory or really conservative solution, if that was going in the direction of the question.
speaker-0: Yeah, okay. Thanks.
speaker-1: Beautiful. Thank you. Hearing us. any other questions already? Okay, perfect. So I get to ask more questions. Thank you guys.
speaker-0: Yes.
speaker-1: Yes, something I wanted to ask you, because I get that question a lot also, and obviously it's changed the way we work a lot, is how do you use AI tools in your own in your own work right now? And Paul, especially I know you you teach, you have students, how do you handle that like and how do you what's your advice to your student on how to use these tools? And then Steve, I'm definitely also super curious. AI tools, like in general. AI. AI tools. Yeah. Sorry. Would you like to start?
speaker-2: yeah, totally. so s I use AI stuff all the time. I learned recently I was like the second largest token user at the Flatiron Institute. yeah. I love
speaker-1: Yeah? Second one, who is the first one?
speaker-2: I will not disclose their name.
speaker-0: Yeah, that's good to go.
speaker-2: I'm fine saying, yeah, I I don't I like AI, it's cool, it's
speaker-1: Yeah. Yeah, no, I know. I d but my question is not do you like it or not? It's like how do you use it best? You know, because it it's a tool so
speaker-2: It's nice. So sometimes like if I'm like working on how do you say, I will use it for like the initial review pass or like to help me with a review. When someone submits code, I will usually have it do a first pass look to say, well, what seems weird here? I will often be the person who like so I am often sketching out the top level API. and I will kind of treat like a vibe coding session where like but by vibe coding I mean literally like I am coding and then I go, Hey, like this is what I want this to look like. Just fix all the syntax errors. And then it goes through and it fixes all my syntax errors for me, which is Super nice. Codex in particular is like weirdly good at the Stan Math library. And yeah, it's just it's really nice. Like if I generally need to write a new function with a gradient, I will I have a little skill that I'm gonna be putting up a pull request for soon, which literally like gives it access to things like simpy to like be able to like find the gradient calculation and like do the prim and reverse mode. Or like the regular regular function and then the reverse mode function. and so like I think I use it quite a lot for like how do you say just speeding up myself. Like it's not like it can do all of my job for me, but it can do it can give me like a twenty percent boost in my productivity. So I think that's those are the main ways that I'm using it right now.
speaker-1: Yeah, awesome. I think that that will be very helpful to listeners. Paul?
speaker-3: I also often use it for example for vacation planning, it's very useful. Different goals and then it writes some nice summary and it tells you of course you made a great choice. But also sometimes I just forget some stats, machine learning, whatever concepts and then I just ask you can you remind me of the math or like having this kind of rough idea of how things should relate to each other, can you just give me a thought and then know can give me some math or some papers or whatever and of course I have to verify sometimes it's garbage, sometimes it overlooks something but often it's kind of you know going in the right direction so I don't have to look things up and then I can only look things up that I'm unsure about so that's very useful but of course you know we have been here growing up kind of without AI so we had to learn a lot of stuff. ourselves so now hopefully we are skilled enough to verify what AI gives us so it's a speed improvement but doesn't make us stupid or replaces us. But what you asked about students, so I recently had a course about a patient data analysis which essentially stand in BRMS and I'm giving another course in the other semester about simulation based inference so I'm always ending up with a project where people have like a data analysis project essentially with the methods we teach and then they are giving a presentation graded and a report which is And you need to pass both presentations first and then report a second. And I have seen like lately of course that the reports are getting less and less valuable to the degree that I skip over most parts of the reports. Nobody cares about the introduction. I can write this so I'm just checking very small things. But what I do is like basically, you know, check the presentation in a way that you know, you kind of have to make an oral accent with them because the rest AI can, on the level of student projects, can do very well. So what happens is a lot of students, they don't really show up to the course, they don't really show up to the question session for the project. they just come, give an okay presentation, which is clearly mostly iGenerated. Their code is clearly iGenerated. Even the way they talk is okay, it's not iGenerated, but sometimes they're reading from an iGenerated text, they're like, can you please give me your phone, you know, I would like to give you a presentation. But then when you ask them... basic questions, which if they had done it themselves, like just failed once or twice themselves, wouldn't trivially know. They have absolutely no clue. Let's say ask what kind of neural architecture does he use for your API model, which he wouldn't know if you've written the code. And it's fine that AI writes it for them, but they just. didn't learn anything, so I let them fail. But it's really sad to see because they're replacing themselves with AI, so they're making themselves useless, which I'm not sure is the right strategy, because if you're only as good as the AI tool that you're using, why would any company or professor or whoever hire you? So I'm trying to teach people, you need to understand it themselves and then just use it as an assistant rather than the assistant taking over your thinking. But the other perspective on this is I'm getting old and I'm getting conservative, so this is like new and strange and scary and perhaps I'm just not quick enough to catch up on the reality.
speaker-2: Nice. Like several asterisk wear old. So take it the U
speaker-0: Yeah.
speaker-3: Execution.
speaker-2: I will say like I for kind of like I another way that I think about it is that like we had Compilers like ten years ago, like you could not sometimes trust the things they would do. like a compiler, internal compiler error ten years ago is not a crazy thing to see. and we're kind of in this weird phase where it's well like you can just kind of now dictate what's happens and the AI is kind of a you kinda think about like a compiler. we're not really at a point where like we can like totally trust the compiler, right? and it becomes also another weird thing where it's like, well, like are these skills that we're gonna need? Like I don't I don't know. I don't have a strong opinion about it either way, but I do find Overall nice.
speaker-1: Yeah. Yeah. No. yeah, I resonate with a lot of what you just mentioned here. We don't have a lot of time left, so I won't I won't dive into that. But last opportunity for questions, Sean? Here you go. I think you already have a sticker, right? You already have a sticker, I think, or you're being greedy and you want a second one. I don't have s a sticker. I don't think it's okay. so actually this this theme about you know when you have an AI or a software, right, more generally that that gets it done for you. Like I I feel like we've been having this conversation about Stan and BRMS and other products around it for a long time, right? So the fact that you know Stan abstracts the inference, like should we be concerned that some of the users don't understand the algorithms that underlie Stan, whether it's MCMC or variational inference. And same with BRMS, is the abstraction maybe preventing people from understanding the the model. Well enough. How can we think about the balance that needs to be struck for different users? Do we really always want to make it as easy as possible for the users, or is it sometimes useful to to challenge them and and and force them to think harder?
speaker-2: I think personally kind of the way I think about it is like, well, like we were having when we were having lunch the other day, we were talking about like someone who was teaching students and it's like, well, like one of them just used BRMS and they were like, This is fantastic. Like I kind of only teach them like a very basic machinations of like how the low-level stuff works. I don't really make them look at Stan itself. and there was someone else who was like, Well, I really want them to understand what happens, right? Like I think it's just like vitally important. I kind of again kind of going back to like computer stuff, I can r I used re you know, twenty years ago you'd really know how to write assembly. Like it was like very important if you know how like read and write assembly. And that's just way less now. Like now there's like things like Godbolt that lets you just read your assembly from your compiler, but like you don't have to like actually write it yourself. And I think that like that's kind of analog that's kind of like an analogy to like what I how I kind of feel now, where it's like, well, like it's good to be able to read this stuff. Do you always need to be able to know how to write all of it? Maybe not.
speaker-3: Yeah, you are definitely younger than us
speaker-1: Yeah.
speaker-3: in that perspective. No, I mean, I always, I don't have a good answer to this, but other than thinking, of course, you know, it's always a question of what, if it's kind of the AI stuff that we're seeing now, is it, definitely something qualitatively different, right? But does it fundamentally change the game of putting abstractions over abstractions over abstractions? Like, is it just like another step quantitatively or is it something qualitatively? I don't know, right? also cannot know because of course it's kind of something that came up when I was like already over 30. So it's... I am not entirely sure to what degree students need to understand, but of course I'm in the stats department now, so there's a certain expectation that they would need to understand certain lower level stuff. If I were to teach psychology students, I would just need them to understand what you put in, roughly what the model does, what you put out, can you trust it, can you not trust it, how to interpret it right. I never teach an MCMC sempler, like just roughly how it roughly works, perhaps metropolis. So I think it depends essentially on the audience. But then perhaps also my expectations are just strict because the people we are now having as students, they will go to industry, they will build some databases or some analysis pipelines. They will probably never ever touch base even again, right? So I'm not really sure if I'm just getting too conservative and too... I don't know, strict with what I expect people to know, or if I have a valid point.
speaker-1: Amazing. Well, last question for you two. I usually ask two questions to end the show, but I don't wanna put everybody on hold. So just one for each of you. quick answer. Steve, if you could have dinner with any great scientific mind, dead, alive or fictional, who would it be?
speaker-2: it would be Jib Simons, who started the Flatiron Institute. I got to meet him twice and he was one of the most charming people I've ever met and he was just a very I'd be very happy to see him again I guess.
speaker-1: Okay, great. Short answer. I appreciate that. And I think you're the first one to answer that, so well done. Paul, you already answered the two questions when you you came on the show a few years ago. So since you like to travel, I'm gonna ask you what would be your ideal destination for next stand? Like you have carte blanche.
speaker-3: I think I would like to have an island near the equator where you have beautiful beach but also still intact coral reefs or restructured coral reefs and we could just go have a beautiful conference and then all go to the beach and enjoy nature while it lasts. think that... Yes.
speaker-1: Yeah.
speaker-2: Stancon Puerto Rico.
speaker-3: That would make me very happy.
speaker-1: Yeah, okay, awesome. Yeah. So sounds like we're going to the Galapagos, guys, next time. So thank you so much. thank you so much everyone for being here, everybody who is listening right now. Obviously, thank you so much Paul and Steve for taking the time and being on this show. And of course please big round of applause for the organization committee, especially Sarah Hamis, Mons Magnuson, sorry for the pronunciation, and obviously also Aki Vetteri who all invited the podcast here and did a tremendous job. Thank you so much. This has been another episode of Learning Bayesian Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbasetats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's LearnBaseTats.com. Our theme music is GoodBay by Beba Brinkman. Fit MCLAS and Mega Rand. Check out his awesome work at Bebabrinkman.com. I'm your host. Alexandora. You can follow me on Twitter at Alex underscore Andora like the country. You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbase tax. Thank you so much for listening and for your support. You're truly a
speaker-0: Good basie and change your predictions after taking information And if you're thinking I'll be less than amazing Let's adjust those expectations Let me show you how to be a good daisie Change calculations after taking fresh data Those predictions that your brain is made Let's get on a solid fountain
speaker-2: Shit.