For prospective randomized studies when alpha is split… This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.drvinayprasad.com/subscribe
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_01Oh, look at this question. What do you think about journals publishing figures in confidence intervals, with 95% confidence interval, when very often the alpha is split and may give the appearance of significance even in a negative study? So explain some of that for the audience. Yeah. That's a good point.
SPEAKER_00When I say for the audience, I really mean explain it for me.
SPEAKER_01All right. So, you know, a lot of papers published the 95% confidence interval of the difference between the two arms for some outcome, which basically says that, you know, 95% of the times that you would run a study, you would get a difference that falls within this bounded estimate or the true value exists in this confidence interval 95% of the times. And a poor man's way of judging statistical significance is if
SPEAKER_01the 95% confidence interval does not overlap one or no.
SPEAKER_00One is no, meaning it could go either way.
SPEAKER_01Yeah, one meaning that there's no difference observed. If the 95% confidence interval does not cross one, that is a poor man's way of saying this is a statistically significant, maybe not meaningful, but it is a real difference that's a phenomenon of the intervention and not chance alone.
SPEAKER_0095% probability that's the case.
SPEAKER_01Oh, you know, it's very tricky wording. I think I even misspoke. What it is is that the true difference exists in the confidence interval 95% of the time. But in this particular instance, the difference may be 100% in or 100%. It's either there or not. But the true difference is not. And I think one has to be very careful. These statistical purists will come in. And hang you for misspeaking it. But for all practical purposes, what you're saying and what it is, is like a fair understanding.
SPEAKER_00Okay.
SPEAKER_01Okay, so does the conference interval cross one? But then what this person is saying is noting, The way they run a study is, at the outset of the study, you have to decide what is the thing you're going to be looking at. What is your pre-specified primary endpoint? And you're budgeted a certain amount of alpha error for that. That's the alpha, which is the probability you would see a difference in that primary endpoint even if no difference exists. And that's typically set at .05. So p-value of .05 means that, boy, if I just stuck my hand in the same jar of M&Ms twice
SPEAKER_01and I counted how many yellows I got, 5% of the time you would see a difference this extreme or more. That's what we're calibrating for. That's the alpha error. But the moment you have a split alpha, which means that there are multiple endpoints the authors have designated as alpha, you have to take that 5% and break it out. 2% for, you know, 2% for 100%. So this happens in like a co-primary endpoint. The primary endpoint is heart attacks or death from all cause. The overall alpha of seeing any difference when no difference exists is 5%,
SPEAKER_01but there's 2.5% for each of these. So now you'll see that the 95% confidence interval won't – will give a false view of the – if it doesn't cross one, that could be true even when the trial is not statistically significant by the pre-specified statistical plan.
SPEAKER_00So I don't know who this person is asking this query. It's a good question. It's AHXM70N. So they're an alias.
SPEAKER_01Yeah, but it's quite a detailed question. Okay. So then, so then to this person's point is it is entirely possible that if alpha is split many ways and the other way alpha split is not just between endpoints, but between the number of times you look at something. If you look at it, you say, we're gonna have three looks at the data. You're splitting alpha across the looks. Okay, so if the alpha is split in many ways, a p-value of 0.049 may not be significant for any endpoint, but merely visually showing the 95% conference intervals will make people think
SPEAKER_01that it's significant. It doesn't cross one. And one more extreme caveat. In certain situations, it's entirely possible that the 95% confidence interval crosses one or touches one, but it is still statistically significant. There's a slight discordance around these edge cases. Okay. So what do I think about them doing that? And any other thing that I have to say is that I have to say that I have to say that I have to say that I have to say that I have to say that I have to say that I have to say that
SPEAKER_01Your visual representation takes me away from that purity. It's more likely to lead to misleading conclusions, etc. I'm also a realist and understand that fleetingly few things we do in medicine reach that level of purity. And we're all trying to explain data with these sort of complicated stories we tell ourselves. And looking at these confidence intervals sometimes helps us come up with those kind of stories.
SPEAKER_00The narratives that we tell.
SPEAKER_01But then my final answer, I think journals are fundamentally in the business of raising their impact factor and revenue and not the scientific truth. And so I have a lot of problems and this is along the lines of the kinds of things I have problems with.
SPEAKER_00Yeah. Yeah. Beautiful. Let me ask a really dumb question. How do you even determine that alpha in the first place for each endpoint?
SPEAKER_01Um, okay. Is that a statistics thing? No. I mean, I guess the overall alpha of study is convention. And typically we have said since R.A. Fisher and the dawn of statistics that P of 0.05 is nominal significant. Right. You know, there's no, there's no more than a one in 20 chance. This will occur when chance alone happens. Chance alone. Right. Okay. So, but that's an arbitrary thing. That was just the original dude picked it. Yeah. And a lot of early phase studies, phase two studies, except higher alphas, 10% alpha, um, uh, you know, um, You know, a 10% alpha is accepted in some studies.
SPEAKER_01Is that because they're smaller studies? And maybe the goal isn't regulatory approval, but the goal is just to get a sense of is this working or not? Right. How you split the alpha is entirely up to you.
SPEAKER_00Okay, when I get my contract as an attending, I'll change my handle. Ah, trainee. Oh, it's a trainee? Look at this, smart trainee.
SPEAKER_01How'd you get in with the moderate admission process being so smart?
SPEAKER_00Whereas my sidekick, Risa, who's a NICU nurse says, I need more coffee to want to try to understand this. To want to try to understand this. That's what I love about Risa. I'm not sure I want to understand this. I love it. My contract isn't attending.
SPEAKER_01What specialty, AH? That's what we need to know.
SPEAKER_00Yeah, you gotta tell us.
SPEAKER_01Because this kind of question, I'd be surprised if it's emergency medicine. My guess would be medical subspecialty. I'd be surprised if it's surgery because the surgeons, they don't bother so much with this. But if it isn't, maybe I'll eat my
Live episode from Youtube This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.drvinayprasad.com/subscribe
Elizabeth Holmes/ AI researchers start smoking/ Moderna vax by Plenary Session This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit...
Pope has cancer; Phase 1 trials for AI; Can AI hack cancer trials__1 by Plenary Session This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit...