Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

In the Interim...

In the Interim... artwork

Adaptive Design Actions Matrix

Jul 27, 202631 minAudio
ShareXLinkedInFacebookWhatsAppEmail

In this episode of "In the Interim…", Dr. Scott Berry challenges the widely held belief that any interim look at trial data obligates an alpha adjustment. By constructing a two-by-two matrix: interim data (positive/negative) and adaptive action (increase/decrease sample size), Scott demonstrates that the need for statistical correction depends on precisely what actions are prespecified. He emphasizes that the need for adjustment depends on the action and the data. Technical scenarios examined include group sequential designs, “promising zone” sample size re-estimation (citing the formal results of Mehta and Pocock), and response adaptive randomization. Scott stresses that clear prespecification is required for Type I error control and regulatory compliance. He critiques common missteps, such as unnecessary allocation of alpha to futility boundaries when superiority is not planned, and reiterates that it is the adaptive action, and not mere data review, that determines the statistical impact of interim analyses. Key Highlights Dissects alpha adjustment myths and their historical roots. Details two-by-two matrix: interim data direction and adaptive action. Explores group sequential, futility, promising zone, and response adaptive examples. Clarifies when Type I error is truly affected - action and data matter. Stresses prespecification’s role in trial validity and regulatory acceptance. Identifies pitfalls in common trial design practices. For more, visit us at https://www.berryconsultants.com/

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Judith: Welcome to Berry's In the Interim podcast, where we explore the cutting edge of innovative clinical trial design for the pharmaceutical and medical industries, and so much more. Let's dive in. Well, welcome everybody back to In The Interim. I'm your host, Scott Berry. Appreciate you joining me again today. Uh, In The Interim is a podcast of all things clinical trial science. Now, if you go back to some of this, I, I try, I, I try to create a, a,

a much more welcome intro to each of the topics, and it might be a story of something that happens that I feel like is relevant to introduce the story. So I, I was, I was talking with, uh, uh, my family, I was talking to my wife, Tammy, who she ends up being a fairly common topic on this, this podcast. And I was describing the, the topic and thinking about is, you know, is there something, there's gotta be something where, um, r- really the topic is

where something is true in a particular scenario and people understand that, but it's not true in another scenario, but they carry that with them and there's a sort of myth about it that this is a, a much more universal thing. And, um, and not understanding the aspect of it, and this just carries forward. And was, was talking about superstitions, for example. Uh, we were… I was, I was talking about we sit at our, our son's baseball game, and we

might not say a word at the baseball game, but then when we're home at the TV, we might not say a word thinking that, you know, th- th- this is different. So I was introducing this, we were thinking about many of this, and I'll introduce exactly what I mean by this. And I said, "Well, you know what, what I'm trying to describe is there's this myth in clinical trials that looking at data Causes penalties. And, uh, I'll, I'll sort of explain more of this.

And there, there's this, this thought that, uh, uh… And it is true in some scenarios. In some actions you're going to take in the trial, you have to adjust alpha because of those actions. But it depends on the action you're gonna take. And in other settings, you, you could do a million futility analyses and there's no adjustment to alpha through, through it. But there's this perception that looking at data causes penalties

of, you know, what, what's a, a common thing in asking her ways that I could introduce this topic. And she, she was thinking about ways, and then she said to me, "Haven't you done that already on your podcast? Don't you do that, you know, multiple times? Are you gonna talk about that again?" So, uh, and it is true. And, but it… I was triggered again today. I was doing a, a call with a potential new client, and they brought up the

idea in their trial, uh, enrollment is slow and, and various issues. I won't go into it. I don't, I don't want to give away any confidential information. But the topic came up that they were thinking about doing futility analysis in the trial, and they didn't wanna do it because they thought they would have to adjust alpha. And this overall perception largely that looking at data is a bad thing to do, that that sort of i- is a corollary to that, and it comes to this.

This is almost an everyday occurrence. And so when my wife was trying to come up with me with, with ways to introduce this topic, in a way, it actually… The act of trying to come up with a story is my intro to the topic today, where, uh, uh, you know, "Haven't you done this before?" Uh, and I have done this before. It's a little bit different today. I'm gonna come at this in a different way today. But I wanna get at this topic and… of, of adaptive actions in trials and what that means.

And there's actually beautiful mathematical aspects to this, so I wanna get into that. The adaptive action matrix in trials, and it's really, it, it's really kind of a beautiful thing, and I don't think it's widely understood, that aspect of it. So let's back up a little bit and think about how does this myth Uh, uh, come about and why do people still believe that looking at data somehow is a penalty i-i-in the trial?

Historically, really the only adaptations that were done in trials were to look for superiority. Group sequential designs. So at seventy-five percent or fifty percent of the way through the trial, we would do an interim analysis, and if the data were promising enough, and we would have to adjust alpha for that because you can't do a, a two point five percent test multiple times and still have two point five percent error in a superiority test.

And so we have to adjust alpha for the multiple looks. That was really the only adaptations that were done. Simple designs in this, and we had the really beautiful mathematics of alpha spending functions, whether it's, it's Pocock or Brian Fleming, Kim De Mets, whatever spending function. We had this beautiful math around it, th-this adjustment. And so these are the only times it was done. And so we'd be running large phase three trials, and this is it.

And with that came the notion that, okay, every time we look at data, you have to make an adjustment. And it's not true. When you do an interim analysis for superiority, you have to make an adjustment. And those were the only times it was done. And so there was this general perception that, okay, every time you do that. And it got attached to looking at data, not the adaptive action of claiming superiority at the time, which does involve an alpha adjustment.

And hence it's still ubiquitous, and I, I, I heard it today, and they were surprised when I told them, "Oh, you could do lots of i-interims for futility and no adjustment needed, uh, in that setting." "Oh, that's great. We, we should do that." Yeah, you should do that, um, i-in the setting. And so it's this, this, this notion that looking at data means you need to adjust. There's penalties to it. It's bad to do it.

That's the other corollary to do that, that it's bad to do it. So Tammy is right. My wife is right. I did a podcast on this, and she even says something like spending alpha. So episode sixteen was spending alpha, and, uh, it was, last year it was the most listened to, watched podcast that I did somewhat for a Bayesian statistician. Spending alpha was one of the more common ones. Um, so you may be interested in going back to that. I want to talk about something a little bit different, uh, in that,

and that is the different adaptive actions and potential implications There's a really, really cool aspect to this, and I'll, I'll go into it. And, and so I want you to envision a matrix. I want you to envision the matrix where the columns… Uh, by the way, one fun thing of doing a podcast, and it's a really good, um, activity to have to do, is to do this podcast knowing people are

listening, and I can't use slides. It's, uh, we, we use slides become a crutch, a graphic around that, and they're wonderful. Uh, if ability to it. But the ability to have to explain something without slides, I think is a, is a good activity to go through in thinking about how to communicate ideas. Uh, it, it makes me a beg- better communicator. I don't know about a good communicator, but I, I think it's helpful. So I want you to think about the columns here as the data.

So we're gonna do an interim analysis, and the columns are positive data. And I'll come back to what does it mean to be positive data. So when the data are positive is the right-hand column. The left-hand column is when the data are negative Okay. And so I'm gonna differentiate when you do an interim when the data are in these cases.

And then the, the rows here are the action, and almost every action you take at a, at an interim has an implication to the effective sample size in your trial. And think about it that you increase the sample size or you decrease the sample size at the interim. And I'll talk at that it's not explicitly that you change N.

Sometimes you take other actions that have an implication to your sample size. So here's our two-by-two matrix. The columns are positive data on the right, negative data on the left. The rows on top are increased sample size, and on the bottom, you decrease sample size. So it's this two-by-two matrix. The really cool thing about adaptive actions and do we need to adjust

alpha, does it cause the need to adjust alpha, is depends on which of those four quadrants you're in. So we talked about the historical scenario where when your data are good, you decrease the sample size. So if we're doing a standard group sequential trial where a hundred is your maximum sample size, this is a generic number, and you do an interim

at fifty patients or seventy-five patients, let's do fifty patients, and we look to see if the data are, uh, below a adjusted P value, .001. If it is, so fifty patients is when our data are really good, we decrease from a hundred to fifty. That's the quadrant where our data are positive. We decrease the sample size. You have to adjust for actions in that quadrant.

You are inflating type one error when you are in that quadrant, and that's the one that every adaptive design lived in that quadrant. And everybody thought, "Okay, everything in all three quadrants, you have to adjust alpha," but just by looking at data, and that's not the case. So, uh, in that, in that, uh, quadrant, the most common one of these, of course, is group sequential designs.

And yes, you have to adjust alpha for that Okay. And we, we, we have other quadrants. So let's think about the other most common quadrant is when your data are negative and you decrease sample size So, and we all understand that that's futility. So when your data are negative and you're gonna decrease sample size,

you actually deflate type I error. You take away the shot that you might have won had you gone to 100. So at 50 patients, when our data aren't very good, we stop for futility unlikely, and the trial loses. You only lessen the chance of success, and under the null hypothesis, you lessen the chance of getting an incorrect positive result. That's type I error.

Um, in that case, we lessen it. We actually deflate type I error. So in this quadrant, the lower right, when the data are positive and we decrease the sample size, we increase type I error. But when you move over a quadrant to the left of that, when the, when the data are negative and y- your action is to decrease the sample size, you deflate type I error In that setting. Now, there's-- it's controversial as to whether by doing futility,

we get to actually buy back alpha, and that a, a, a different topic for a different, different day. Um, uh, in that setting, generally, that, that's generally not done, but in some s- cases, it might be done. But that's, that's the result of it. So there's a circumstance where the advice I gave today was you can do interims for futility over and over and over and over again, and you can't inflate type I error because the only action you can take in that scenario is to decrease the

sample size when the data are negative. Now, the important thing is, and I, I, I described that we would talk about, what does it mean for the data to be positive or negative? Now, it can depend by the action, but largely when the data, when the data are being negative, meaning at that point, we do not have superiority. And so that's-- doesn't have to be all that negative. The data can still be on the positive side.

So for these two a- two adaptive actions, stopping early for supe- superiority or stopping for futility, the threshold is actually do we have significance at that time point, uh, within that? I mean, that's kind of the, the, the break point of this as to what data are negative and positive. Fairly straightforward, um, actions in these settings. Now, it gets really interesting when you go to the other two quadrants.

So let's think about the quadrant where the data are positive, and you increase the sample size So, so what, what, what kinds of actions do that? In a phase III trial, an action that does that is the promising zone. So I want to spend a little bit of time on that 'cause I think that's the beauty of the mathematics of it. But other actions at that time are one common adaptive action is

response adaptive randomization. So suppose you have two doses and a control, and we're gonna-- when the data's positive on one of the two doses, we increase the randomization to it. We're increasing its effective sample size. When its data are positive, we increase its effective sample size. You decrease type I error.

In this matrix, the bottom right was, uh, increasing type I error. As you move from bottom right to upper right, you decrease type I error. You're gonna see that in the bottom right, we increase type I error, in the upper left, we increase type I error, and then the two off-diagonals, we decrease type I error. Futility decreases type I error. Response adaptive randomization when the data are positive on the dose, and we increase its allocation for the r- for the next stage or moving

forward, we decrease type I error. Response adaptive randomization decreases type I error. Now, there are details to that about what, what are we, what are we testing in that scenario, but if we're testing are one of the doses better than placebo and we're increasing the rate to that, we decrease type I error. You don't have to i-in-- change your alpha level to do response adaptive randomization a hundred times in a trial. No adjustment needed.

We've had special protocol assessments with the FDA where they agree, and we, we did twenty interim analyses to do response adaptive randomization, and there's no inflation of type I error. error And so that's a really neat thing that you don't necessarily think about in adaptive designs and what are the actions in that. Other things of even dropping arms in a trial. Suppose you have a h- a hundred patients in a trial, and when you get to fifty, you have two doses, we're doing one-to-one-to-one

randomization, and we drop a dose. And the remaining fifty patients, we don't change the overall sample size, the remaining fifty patients that would have been one-to-one-to-one are now one-to-one 'cause we dropped a dose. We increase the sample size on the remaining dose. By dropping that arm, we deflate type I error in that scenario. So r-- and by the way, arm dropping is a form of response adaptive randomization

By making its probability zero in that we're changing its randomization based on the data, uh, in that scenario. So tho- that, that's kind of a weird place within this matrix that we don't spend a lot of time talking about. I wanna go back to the promising zone 'cause it brings out really interesting mathematics of this, where the, the-- what is the promising zone? The promising zone is an, is a adaptive sample size trial that, um, uh,

uh, I'll give you the basics of it. We're going for, we're… for a sample size of a hundred one-to-one randomized trial. The planned sample size is a hundred. We're gonna do an interim before a hundred at fifty, say. And based on the data at fifty, we might continue on to the planned one hundred. But at that time, we might increase the sample size to be bigger than a hundred.

Eh, this is a kind of sample size re-estimation technique, this special one. There's a really beautiful result mathematically, and this comes from Mehta and Pocock paper, where when you do the look at fifty, if the conditional power for a hundred is above 50 You can increase the sample size above a hundred to anything you want,

and you don't increase type I error. You actually decrease type I error So that's the beautiful result where I describe-- I, I said when the data are positive, you increase the sample size. This mathematical result, so when you're sitting there at 50, if your data are trending to winning at 100, conditional power of 50 means you're right on the ef- the effect size you need to win at 100.

If you continue on with that effect size, you're gonna win. If you get a little bit worse, you're gonna lose. Anything better, you're gonna win. That's conditional power of 50. Right now, we're at the threshold we need to win. That's mathematically the definition of positive data when you're sitting at 50 and you're planning 100. You could increase it to anything you want. So this design allows you, based on as long as your conditional power is above 50, you can increase your sample size to get 80% power.

You can increase it to 200 to get 80% power within the scenario. Notice a couple things of this. In this matrix of positive data and increasing your sample size, if you're right on the line of positive data, so you're at 50% conditional power, you don't lose any alpha in that. But if your conditional power is above that, you're actually now spending alpha. I, I'm sorry, you're losing alpha, like futility, that the promising

zone design loses alpha, uh, i- in that because of this action. Because at 50, you're increasing to a sample size, but you can only do it in a very narrow region, uh, of the design. So a separate podcast, my Tammy may be listening at this and say, "Ooh, I've heard this somewhere before." By the way, she does listen to each of the podcasts. Um, I, I did a podcast, episode 31, the not so promising zone

design It's, it's commonly done, and it doesn't work very well. So please go into that one. So the math is beautiful, fantastic. The design and the actions are not. It doesn't work very well. If you compare it to other adaptive approaches, it doesn't work very well. Spend alpha and do a group sequential. That's the spending alpha episode 16. So it all kind of ties in together.

I got… You know, so I don't wanna go deep into those. Please join those episodes if, if, if any of that is interesting. So the promising zone design lives in this new quadrant that we haven't been doing many adaptive designs in there. Now, the, the interesting thing is they've created ways that you can solve exactly. Sometimes you can push that fifty percent even lower. You know, forty percent conditional power thirty-seven without inflating type one error, uh, in that scenario.

So you can calculate the math, and then if you go below that, where your data is defined to be negative, we actually move the quadrant from the upper right to the upper left, where our data at that time are negative, and we increase the sample size. That increases type one error. So now we've got this matrix of actions and data. And re-remind us where we are. The, the, the right-hand column is positive data.

If you decrease the sample size or you do an action that decreases sample size, you increase type one error If you do something that increases the sample size when you have good data, that decreases the type one error in those several actions. Moving over to the left-hand column are negative data. And so now to win at 100, we're actually in this conditional power less than 50%. We're, we're, we- we're, we're not trending to a successful trial.

Our data are negative. What if we wanna push the sample size to 200 or, or to 1,000? Now, when the data are negative, if you increase your sample size, you're almost saying, "Oh, just give me a new trial. Give me 1,000 patients. Give me bigger numbers." You're almost taking multiple shots on goal by doing that. When it's negative, I'll just collect enough data that will, will, will, will weigh out this initial negative. You're, you're doing kind of the opposite of group sequential, but you have the same effect of it, that you're getting multiple shots on goal.

You increase type one error by doing that. This is not understand- stood very well that the, the-- this circumstance. There are a lot of people that I see designs where, oh, we'll just do an interim at the design, and we'll push the sample size to be whatever sample size gives us ninety percent conditional power. That trial will inflate type one error i- in the scenario. You, you, you don't get this free lunch.

So if you're, if you're stuck on type one error, you live in that quadrant there of doing that. A group sequential design, you could almost think of group sequential as the opposite of we have 100 and we might stop at 50. A group sequential design is we're gonna go to 50 and see if we win. If we don't win, we're gonna increase the sample size to 75 and see if we win. And oh, if our data are negative, we're gonna go to 100 and look at it. You can rewrite a group sequential design to live in this quadrant in the upper

left, where we're taking actions on negative data to increase our sample size. Multiple shots on goal, you're doing things that increase type one error Okay. So now this is the, the, this is the circumstance. Now overall, the general understanding of this is because we're in the lower right-hand quadrant, that looking at data is bad. There are huge tools throughout here to create a better design, and going back to

this episode, uh, 16 on spending alpha, spending alpha is a good thing to do. Recognizing your, your actions here and when it happens in the data in order to create a better design to get better answers within the constraints of it is a good thing to do. So I hope envisioning all of this as your actions, the important part of it is looking at data doesn't increase type one error.

It's the action you may take. Really important part of adaptive designs, this is why it's absolutely critical you have a prospective adaptive design. You get a great deal of flexibility in the design as long as it's pre-specified. If you try to go with an adaptive design as, "Oh, I'll just look at the data and then decide what to do," you can't argue you're in this quadrant, you're in that quadrant, these are the actions I'm taking.

You can't even control type one error from such a design because you haven't defined the actions at that time from it The other, the other thing that I, I find particularly annoying, uh, my pet peeve of the day, Kurt Veily has pet peeves of the day, is I'll see where somebody's doing a futility analysis, and they spend .0001 alpha as a-- But are you gonna do a superiority test?

No, no, we don't wanna do superiority at all. But they're throwing alpha to the gods because they almost think they have to. And it's almost disingenuous to do that because you don't wanna talk for-- stop for superiority, but yet you allocate it as though you might do something else. What, what, what's the design? I think it's a circumstance where you just be very clear, be prospective. You can have a great deal of flexibility. You can design better trials by being fully prospective in it.

A great deal of flexibility in the sample size. Uh, you can accomplish what you're trying to accomplish with, with efficient trial designs, but it has to be prospectively set up, and you can take advantage of some of the math here. Um, going back to introducing this that, uh, Tammy may say, "I've heard all that before." Uh, hoping this matrix idea of thinking about it, data positive, data negative, and the action being increasing effective sample size or decreasing effective sample

size might be something new for you to think about the world of adaptive designs. I, I was going to go into this whole aspect of in golf, knowing the rules in golf, you can actually drop your ball in certain situations. You get a great deal of flexibility under the rules of golf. Um, and it's not always you can't touch your ball, you can't do it. Knowing the rules helps you to be a better golfer. I was gonna enter with that, but I thought I would enter with, with my

wife saying, "You've done this before." And yes, I've done it before, and it's in part because every day in my day job, I hear these kinds of things from statisticians and clinical trialists alike. So know the rules, design better trials, use them to your advantage. Efficient, good trial designs out there. And until the meantime, we will be here, maybe spending alpha, maybe not spending

alpha, but we'll be here in the interim.

Open the transcript file

Hosts and Guests

Hosts

  • Scott Berry Host Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can Borrowing Ever Increase Power?

    Nov 16, 202625 minAudio

    In this episode of "In the Interim…", Dr. Kert Viele, Director of Research at Berry Consultants, takes up a requested topic: published results showing that Bayesian borrowing of historical control data cannot increase...

    Credit the people on this episode

  2. Mailbag: Learning Healthcare, Multiplicities, and the Pace of Innovation

    Nov 9, 202647 minAudio

    In this mailbag episode of "In the Interim…", Dr. Scott Berry answers two listener questions and weighs in on the pace of innovation in clinical trial science. Asked what infrastructure a learning healthcare system...

    Credit the people on this episode

  3. Reflecting on EMA Reflections on Platform Trials

    Nov 2, 202644 minAudio

    In this episode of "In the Interim…", Dr. Scott Berry and Dr. Kert Viele, Director of Research at Berry Consultants, review the European Medicines Agency's draft reflection paper on methodological aspects in...

    Credit the people on this episode

  4. Statistical Methodology Groups in the Pharmaceutical Industry

    Oct 19, 202656 minAudio

    In this episode of "In the Interim…", Scott Berry is joined by Jenny Devenport and Kaspar Rufibach of Merck Group, co-authors of the EFSPI Statistical Methodology Leaders paper "Statistical Methodology Groups in the...

    Credit the people on this episode

  5. A Discussion on Virtual Controls

    Oct 12, 202647 minAudio

    In this episode of "In the Interim…", Dr. Scott Berry shares the framework he presented at a CSRC regulatory science hackathon on digital twins and synthetic controls in cardiovascular clinical trials. He separates...

    Credit the people on this episode