[Kristin] (0:00 - 0:12) These two separate experiments both seem to have faked data in them in the same paper about honesty. It's like we have two independent cases of dishonesty in a paper on honesty, which is pretty hilarious.
[Regina] (0:13 - 0:14) You really can't make it up.
[Kristin] (0:20 - 0:29) Welcome to Normal Curves. This is a podcast for anyone who wants to learn about scientific studies and the statistics behind them. I'm Kristin Sainani.
I'm a professor at Stanford University.
[Regina] (0:29 - 0:35) And I'm Regina Nuzzo. I'm a professor at Gallaudet University and part-time lecturer at Stanford.
[Kristin] (0:36 - 0:41) We are not medical doctors. We are PhDs, so nothing in this podcast should be construed as medical advice.
[Regina] (0:41 - 0:59) Also, this podcast is separate from our day jobs at Stanford and Gallaudet University. Kristin, today we're finally getting into the story behind Francesca Gino. She's the prominent Harvard Business School researcher who recently had her tenure revoked for research misconduct.
[Kristin] (0:59 - 1:47) Yeah, Francesca Gino has had several papers retracted because of data anomalies. And we foreshadowed her case in our previous episode because that case has something in common with today's case. It's the same data sleuths who discovered the problems.
We're talking about the data colada folks. And Regina, the Francesca Gino case is fascinating. It also has a ridiculous number of twists and turns.
And it actually reminds me of this show I was recently watching on Hulu called The Killing. That's a murder mystery, and they keep changing the suspects. So first they make you think it's the kid, and then it's the teacher, and then it's the politician, and then it's the mob.
And it was fun for a while, but then it started to be too unbelievable with too many twists and turns. Funny enough, though, this Francesca Gino case is almost as ridiculous in the number of twists and turns, but it's actually real life.
[Regina] (1:47 - 1:56) Real life. And even better, her research often involved studying honesty, morality, workplace behavior.
[Kristin] (1:56 - 2:15) The irony is rich, and this is why this case got a lot of attention, even in places like The New York Times. So several of the papers she's co-authored have been retracted. There have also been legal battles going on between her and data colada, and between her and Harvard.
And it's still ongoing, hence all the twists and turns.
[Regina] (2:15 - 2:32) And beyond just the general interest, Kristin, it's especially fascinating, I think, to you and me because it involves some of our favorite topics. Statistical sleuthing, research practices, data cleaning, file management, all the goodies.
[Kristin] (2:33 - 2:40) Yeah, those things might seem boring, but all of a sudden they get pretty exciting when they are part of a giant detective story.
[Regina] (2:41 - 3:13) Right. So in this podcast, of course, we always have a claim to evaluate. Usually it's the research claim of the paper we're covering.
But since this paper has already been retracted, that doesn't really work so well. So we have to get a little more creative, Kristin. And since in this episode, it's an Excel spreadsheet that has a starring role, actually, how about this for our claim?
Excel spreadsheets are neither sexy nor reliable data management tools.
[Kristin] (3:13 - 3:21) I like it. And I think this is not the first time, Regina, in this podcast when Excel has been somewhat the villain of the story.
[Regina] (3:23 - 3:33) That is a very good point. Spreadsheet. The new villain.
Okay. So at this point, I also need to point out that you and I know nothing about the law.
[Kristin] (3:33 - 3:34) Yeah, zero.
[Regina] (3:35 - 3:40) So I am going to make sure that we are extra careful in this episode.
[Kristin] (3:40 - 3:47) Yes, we are going to stay in our lane because we have no legal expertise and we're just going to comment on the data shenanigans.
[Regina] (3:47 - 3:49) This is a pretty high stakes case.
[Kristin] (3:50 - 4:06) It is. Gino is being advised by Lawrence Lessig, a famous Harvard law professor who has produced an entire podcast series defending her. And also, Gino sued the three Data Colada researchers for $25 million.
[Regina] (4:07 - 4:16) And Kristin, you and I do not have that kind of pocket change on us or in the sofa cushion. So I guess we should be careful, huh?
[Kristin] (4:17 - 4:19) I think we need to be careful. Let's not get sued, Regina. Yes.
[Regina] (4:21 - 4:37) Good plan. Good plan. Okay.
The whole thing is also pretty sprawling. So Harvard's generated something like 1,300 pages of material. Lessig devoted nine podcast episodes to it.
[Kristin] (4:38 - 4:52) Okay, Regina, we are not going to do nine episodes on this, but we are going to split this up into two episodes. So in the next episode, I'm going to examine another one of Gino's disputed studies and I'm going to pick apart some of Lessig's arguments defending her.
[Regina] (4:52 - 5:43) But today, we are just going to follow the data detective. So we're going to look at one of the four experiments that raised concerns. And the experiment was interesting.
It kind of just asked a simple research question. Can you make people more honest just by changing where on the page they sign a form? And of course, the answer wasn't quite so simple.
Otherwise, we wouldn't be here in the first place. So our honesty fraud saga actually starts, Kristin, 14 years ago, 2012. And that was when this experiment was published in the Proceedings of the National Academy of Sciences, co-authored by five researchers.
And that included Gino and Dan Ariely, who we talked about in our last episode, and then three other researchers.
[Kristin] (5:43 - 5:49) And Regina, this is actually an experiment that has been used in practice, right? People have actually applied this result in real life.
[Regina] (5:50 - 6:28) They have. Governments have spent money to implement these results. They have changed official forms because of this.
Now, the published paper that we're talking about actually talked about three experiments. We're just going to focus on the first one. That was the one conducted by Gino at the University of North Carolina shortly before she moved to Harvard.
And the setup was this. Participants came into the lab, and they got a worksheet with 20 math puzzles. They had five minutes to solve as many as they could, and they would get a dollar for each puzzle they reported they solved correctly.
[Kristin] (6:29 - 6:33) Oh, wow. So they could get up to $20. And you know, Regina, that sounds actually really fun.
I want to sign up for that study.
[Regina] (6:34 - 6:49) I was actually thinking when I was reading about this that I should make a math puzzle test for you because it involved doing a lot of three-digit addition in your head really quickly. And I feel like you could do this in your sleep for fun.
[Kristin] (6:50 - 6:56) I think I could do that, yes. Actually, I think that would be really a fun thing for first dates, do math puzzles.
[Regina] (6:57 - 6:59) Kristin, is math your love language?
[Kristin] (7:00 - 7:01) Math is my love language, yes.
[Regina] (7:02 - 7:22) Sexy arithmetic. Okay. But would you be honest, though, Kristin?
Because here's the thing. In the experiment, the participants got to grade themselves. At the end of the five minutes, they counted how many puzzles they'd solved correctly, reported that number, and then they threw away their worksheets.
[Kristin] (7:22 - 7:33) Ah, and since they threw their worksheets away, they thought that nobody could check how many they actually got right. So, in other words, they thought they could cheat without getting caught if they wanted to, you know, up their income.
[Regina] (7:34 - 8:32) That was exactly the idea. But the researchers, of course, had secretly coded each participant's answer sheet so later they could dig them out of the trash and see how much each participant had actually solved, not just reported. Next, the participants had to fill out what looked like a text form, and they reported their earnings from the puzzles along with how much money they'd spent traveling to the lab because they could get partially reimbursed for these expenses.
So, in one version of the form, you signed an honesty declaration at the top before filling everything out.
[Kristin] Oh, is that somewhere where you sign to promise that you're going to be honest, something like that?
[Regina] That's exactly what it was.
They swear that everything was true. In another version of the form, you signed at the bottom after filling out the whole form, and the third version, that was the control. There was no honesty declaration, no signature at all.
[Kristin] (8:32 - 8:34) So, the control is you don't promise to be honest at all.
[Regina] (8:35 - 8:58) Oh, that is a very interesting take on it. You might be right. So, then the researchers compared how honestly people behaved in each of those three groups, and they looked at three outcomes.
One, whether people cheated on the math puzzles. Two, how much they cheated. And three, how much they claimed in travel expenses.
[Kristin] (8:59 - 9:16) Okay, the travel expenses, that one's a little interesting, but I guess the idea is because it's a randomized experiment, they would expect the travel expenses to be pretty balanced between the three groups because, of course, we're not going to know what their actual travel costs were. So, we don't know if they're lying, but an imbalance might suggest that one group is exaggerating.
[Regina] (9:17 - 9:49) That is absolutely the idea. And the results, they were really striking, Kristin. So, in the signature at the top group, only 37% cheated on the math puzzles.
In that no signature group, it was 64%. And the signature at the bottom group, 79% cheated. Wow.
That was a highly statistically significant difference. Signing at the top was significantly better than either alternatives.
[Kristin] (9:49 - 10:09) First of all, Regina, those are all pretty big numbers. Like these participants were not all that honest. Even in the signature at the top group, we still have over a third cheating.
Wow. And the difference between the three groups, that is a really big difference. It's also interesting that more people would cheat when they had to sign at the bottom than when they didn't have to sign at all.
[Regina] (10:09 - 11:05) Excellent observation. I want you to hold on to that thought, Kristin. It will come back.
All right, that was the first one. Now, they also looked at how much people cheated, and the pattern was almost exactly the same. So, people who signed at the top over claimed less than one puzzle, not very much at all on average.
People with no signature over claimed two and a half puzzles. They gave themselves an extra two and a half puzzles that they didn't solve. And people who signed at the bottom over claimed about four puzzles.
Again, signing at the top was significantly better than either alternative. And finally, they looked at those travel expenses. Same pattern.
Top signature group claimed about $5. The no signature group claimed about $8.50, and the bottom signature group claimed about $9.50. So, across all three outcomes, we get the same pattern.
[Kristin] (11:06 - 11:10) Signing at the top looks the best, signing at the bottom looks the worst, and no signature is somewhere in between.
[Regina] (11:10 - 11:24) Exactly. And in all these three outcomes, signing at the top was statistically significantly better than either alternative. But that bottom signature and the no signature groups were not significantly different from each other.
[Kristin] (11:24 - 11:33) Oh, so like the 64% versus the 79%, even though the bottom signature group had more cheating, it wasn't a statistically significant difference.
[Regina] (11:33 - 11:52) Exactly. Perfect. Perfect.
So, the researchers actually summed it all up in one line, which I is actually the headline finding here. Thus, signing before reporting promoted honesty, whereas signing afterwards was the same as not signing at all.
[Kristin] (11:53 - 12:24) Well, that's not really what non-significant means, actually. It does not mean that they're the same. It just means that you didn't find a significant difference.
So, that's a little misleading, bad interpretation of statistics. But that aside, the way they phrased it, it does make for a neat, perfect little story. So, I can see why the press and governments and businesses loved this result.
And it kind of makes sense, right? If you remind people, to be honest, before they report anything, they might have a little guilt in the back of their mind. It makes sense to me that this might be true.
[Regina] (12:24 - 13:30) Exactly. I feel like this is catnip, right? This is one of these psychological findings that we love because it's kind of intuitive.
Once you think about it, it's simple. The effects are huge. And who doesn't want to be more honest and to get other people to be more honest?
Okay. So, it sounds great. It sounds perfect, except we had a few wrinkles here.
We had a failed replication study. And we had the Data Colada folks. Now, Kristin, I love the story of this replication.
I admit it. So, it started when three of the original co-authors of that 2012 paper, so not Gino and not Dan Ariely. So, those three other researchers set out to extend that finding of signing at the top, decreasing dishonesty, right?
And I think they just kind of took it for granted. It would be easy to do a replication in other scenarios, right? A conceptual replication.
And then they could examine the whole phenomenon more deeply. So, Kristin, explain what a conceptual replication is.
[Kristin] (13:31 - 13:51) Right. Conceptual replications are where researchers test the same underlying hypothesis or theoretical concept as in an earlier study, but they use different methods. So, maybe they test it in different conditions or use slightly different measures or experimental setups or populations.
And that's different from a direct replication where you try to copy the original study's methods as much as possible.
[Regina] (13:52 - 14:00) Right. Exactly. Okay.
So, that trio, they tried five different conceptual replications and all five failed.
[Kristin] (14:01 - 14:07) Wow. So, that must have been a surprise, right? We get this strong finding in the original paper and nothing replicates.
Wow.
[Regina] (14:08 - 14:25) Yep. So, when they got to that point, they just abandoned the conceptual replications entirely and they brought the whole gang back together, including now Gino and Ariely, and recruited two additional researchers, and now they set out to conduct a direct replication of that original math puzzle experiment.
[Kristin] (14:25 - 14:32) Oh, so then they tried to actually repeat the exact same thing with the exact math puzzles and the wastebaskets and all of that, right?
[Regina] (14:33 - 15:11) Right. They went back to basics. They're like, okay, we can't do a conceptual one.
Let's just directly replicate this whole thing in the first place. And this time, actually, Kristin, they went big. So, the original experiment in 2012 had just 101 people, but this time, guess how many?
1,200. More than 1,200 participants, yeah. And they did, Kristin, you'll love this, power calculations beforehand.
They pre-registered their analyses. They made all of their materials publicly available. They even included Bayesian analyses in here.
I loved it. Made me so happy to read.
[Kristin] (15:12 - 15:24) We love all of those things. Pre-registration, open data, that's great. And I just want to point out, Regina, direct replication, the one thing you don't have to repeat exactly is the sample size.
You can make it bigger.
[Regina] (15:25 - 16:00) In fact, it's good to make it bigger. Exactly. Okay.
The result of this direct replication, nothing. It failed completely. So, the original study had actually found a large difference in dishonesty between signing at the top, signing at the bottom, like we talked about before.
But in this direct replication, there was pretty much no difference at all. The p-value was 0.62. And Kristin, I think the paper's title gets right to it. I love it.
Signing at the beginning versus at the end does not decrease dishonesty.
[Kristin] (16:01 - 16:10) Oh, great title. It's really direct, gets to the point. And actually, I have to give a lecture in a few weeks about titles and abstracts.
So, I'm going to use that as an example of a good title.
[Regina] (16:10 - 16:39) It's a great title. And the interesting thing was, this was not just a case of having too small a sample, right? Sometimes we see that.
But the researchers designed the study to have 80% statistical power to detect an effect as small as Cohen's d of 0.1. And the original reported effect in 2012 was about 0.7 to 0.9. So, Kristin, just put that into perspective for us.
[Kristin] (16:39 - 17:09) Well, Cohen's d of 0.7 to 0.9 would be considered a large effect size. And so, this replication study was obviously designed to detect even a much smaller, subtler effect. And we talked about power a couple of episodes ago, Regina.
So, if the true effect was really as large as the original paper suggested, this study would have had a really high chance of detecting an effect that big. Even an effect as small as Cohen's d of 0.1, which is really tiny, 80% chance of finding it? That's pretty good.
[Regina] (17:09 - 17:39) Exactly. I loved it. So, I feel like that's pretty compelling evidence that the original finding just wasn't holding up at all.
But here is where the story takes the unexpected turn that did our first twist. The replication was published, PNAS, 2020. And remember how I said the authors made all of their materials public?
Well, that included the data set from the original 2012 paper.
[Kristin] (17:39 - 18:50) Oh, very nice. Oh, and that is how the Data Colada folks got their hands on the data. If I remember correctly, actually, Regina, some other anonymous people noticed some things in that data set when it was released publicly, and they passed it on to Data Colada for a little help. I think that's what happened.
And Regina, it's interesting because this paper had multiple experiments in it, not just the math puzzle experiment. And I think the first blog post that Data Colada published on was actually about an entirely different experiment that involved car insurance. And that experiment, I think, was largely led by Dan Ariely.
Remember, he's a co-author on this And I use this example a lot in class, actually, because Data Colada found pretty obvious data fraud in that experiment. So I can give it to my Stats 101 students and have them uncover the fraud once you know what to look for. It's pretty obvious, because actually somebody randomly generated that data.
And Data Colada published their blog post in August of 2021, this original blog post about the car insurance experiment. That led the paper to be retracted in September of 2021. Pretty quick turnaround, actually.
Usually it doesn't happen that fast.
[Regina] (18:51 - 19:07) Very fast turnaround. But of course, that was not the end of it, because Data Colada also started examining the math puzzle experiment in the same paper that we've been talking about. That was the one associated with Gino, and they found more suspicious patterns.
[Kristin] (19:08 - 19:33) And this is actually pretty statistically implausible, pretty extraordinary, because think about it for a minute. We've got multiple experiments in the same paper that are actually led by different researchers. And these two separate experiments both seem to have faked data in them in the same paper about honesty.
It's like we have two independent cases of dishonesty in a paper on honesty, which is pretty hilarious.
[Regina] (19:34 - 20:01) You really can't make it up. If people wrote this in fiction, you would say, come on, there's no way. So Data Colada did not just stop with the data set with this one paper.
Working with other researchers, they started examining more studies co-authored by Gino. They eventually found concerns with four papers, not just one. And they brought all of these concerns privately to Harvard in 2021.
They didn't publish right away.
[Kristin] (20:01 - 20:13) Yeah, I think this is interesting. They didn't immediately publish any accusations against her. They went to the university to try, I think, to give her a fair shake.
[Regina] (20:14 - 20:44) Right. And the university launched an investigation.
Yeah. Harvard spent about two years investigating. And of course, they had access to research records Data Colada didn't have.
And in 2023, two years later, Harvard investigative committee on this concluded that Gino had committed research misconduct. And that was when Harvard placed her on unpaid administrative leave. And Gino, of course, has denied committing research fraud.
[Kristin] (20:44 - 21:13) Yeah, this was a pretty big deal, actually, Regina, because around this time, Gino was apparently earning about a million dollars a year, which for people who don't know, that's a lot more than most professors make. She was actually one of the most highly paid professors at Harvard. And apparently, she was getting up to like $100,000 per speaking appearance.
And Regina, how do we get those speaking appearances? I'm waiting for the $100,000 speaking appearance fee, Regina. Not there yet.
[Regina] (21:15 - 21:32) Almost. One day. One day.
Okay. So around the same time as all of this, Harvard requested retractions of these papers. And Data Colada, shortly after that, published four blog posts laying out all the evidence that they had uncovered.
[Kristin] (21:33 - 21:52) Right. And this is really when everything went public. And I actually remember when these Data Colada blog posts published, Regina, I was actually on vacation, and I was totally riveted by these blog posts.
I mean, they were really well written and super fascinating. And I was like reading these instead of the fun beach read novel that I had brought along.
[Regina] (21:52 - 22:02) This sounds crazy. And it sounds like that might just be you, Kristin, but it's not. They are.
They are actually fascinating. They're well written. They're clear.
They're even funny.
[Kristin] (22:03 - 22:06) Yeah. You don't have to be a statistician to follow them. They're really good.
[Regina] (22:06 - 22:13) Okay. So now we're up to 2023. Let's talk now about what the Data Colada folks actually found.
[Kristin] (22:13 - 23:03) But let's take a short break first. Welcome back to Normal Curves. We're talking about the allegations of research misconduct involving Harvard researcher Francesca Gino.
And today we're focusing on the famous Honesty Pledge study. Whether signing at the top of a form makes people more honest than signing at the bottom. And Regina, I'm just wondering, maybe we should introduce the Data Colada folks, who they are, instead of just keeping calling them the Data Colada folks, especially since they provided a lot of the material for our last two episodes.
[Regina] (23:05 - 24:04) Okay, that's an excellent point. Let's do it. There are three of them, Yuri Simonsohn, Leif Nelson, and Joe Simmons.
Now, Yuri is a behavioral scientist at a business school in Barcelona, Spain, called ESADE. I don't know. He's also affiliated with Wharton School at University of Pennsylvania and, claim to fame, I actually hung out with him in a beer garden in Munich at a conference a few years ago.
And I had also, before that, interviewed him for my Nature article about p-values, and he is absolutely terrific. Now, Leif Nelson is at UC Berkeley, in your backyard, Kristin, and Joe Simmons is at Wharton. And all three study judgment and decision-making, which is great, and they've done really influential work on p-hacking, research reproducibility, especially in psychology, and I love these guys.
I'm such a fan girl, really.
[Kristin] (24:05 - 24:13) Yeah, I'm a fan as well. They are fabulous. And they are really making a difference in science, getting people to wake up to some of the problems in science.
[Regina] (24:13 - 24:22) Kristin, they're making so much of a difference that they got sued for $25 million because of it. I wish they sold t-shirts. I would buy a t-shirt. I want a t-shirt.
[Kristin] (24:23 - 24:32) I'm surprised they don't sell t-shirts, Regina.
[Regina] They don't.
[Kristin] But, you know, I think that's how you know when you're successful, when famous people start suing you.
[Regina] (24:33 - 24:49) Oh, no. I'm going to leave that type of success to you, Kristin. They can come for you.
Okay, so let's talk about what Yuri, Leif, and Joe found in this whole Honesty Pledge data. And their blog post on this was, of course, called Clusterfake.
[Kristin] (24:50 - 24:51) They are funny. That's a great title.
[Regina] (24:52 - 25:46) They are super funny. Okay, here is the thing. Their first clue was something that was actually kind of boring.
Some rows in the Excel spreadsheet were out of order. Doesn't sound very exciting, right? So, they opened the spreadsheet.
It looked very neat and organized. And the observations, they appeared to be sorted by two things. You know, you could sort in Excel.
So, it looked like they were first sorted by the experimental group. So, all the participants who didn't sign anything, they were at the top. And then below that were all the data from the participants who had signed the form at the top of the tax form.
And then below that, you had those who had signed at the bottom. And then within each of those groups, the participants appeared to be sorted by their ID numbers from smallest to largest.
[Kristin] (25:46 - 26:04) So, for example, within that control, that no signature group, you might see participant 3 and then participant 10 and then participant 13 and so on. They're not necessarily consecutive numbers because, remember, we randomized, but they're always increasing. That implies that somebody sorted in Excel on ID number.
[Regina] (26:05 - 26:07) Uh-huh, except, guess what, Kristin?
[Kristin] (26:08 - 26:21) Regina, were some of those rows out of order? This is actually a trick we've seen before in statistical sleuthing. People sort the data and then they go in and manually change things.
And it's pretty obvious because things become out of order.
[Regina] (26:21 - 26:54) Uh-huh. The spreadsheet remembers. It remembers everything.
It will talk. Okay. So, there were seven observations.
There were seven rows whose participant ID numbers were out of sequence. There was one in that no signature group. There were three in the sign at the top group and three in the sign at the bottom group.
And there were also two observations in the sign at the top group that appeared to be duplicate, just straight up duplicate. So, altogether, nine suspicious observations.
[Kristin] (26:56 - 27:21) If you sort by experimental group first and participant ID second, which is what it looks like happened, those observations should be in order with everything else. Now, of course, we can't rule out that somebody, you know, made a mistake.
And when they were typing in observation 100, they accidentally marked it as a control group when it was supposed to be an experimental group and they moved it later. We can't rule that out, but it is weird.
[Regina] (27:22 - 27:44) It is super weird. And that's why focusing on these weird observations, they asked a very interesting question. They said, if someone had moved these observations around to make the experimental results look better, what would we expect those observations to look like?
[Kristin] (27:44 - 28:01) Right. So, they're trying to determine, is this some kind of random error, again, like I just described, or is this something deliberate? And you can pick out that it might be deliberate if the observations that were moved are unusually favorable to the hypothesis of interest.
[Regina] (28:03 - 28:30) Exactly. And here they were. They were among the most extreme observations in their groups, and they were always in the direction that supported the main findings.
So, for example, the suspicious observations in the sign at the bottom group, they had enormous travel expense claims. It was around $22. But meanwhile, those suspicious observations in the sign at the top group had extremely low travel expenses, around $2.
[Kristin] (28:31 - 28:46) Oh, wow. Those are extreme, because we were talking about the average as being like $5 to $9 before. And, of course, that's exactly in the direction, sign at the bottom having high travel, sign at the top having low travel, that the researchers wanted to see it fits in with the rest of the results of the paper.
[Regina] (28:46 - 29:13) That was a headline result. So, you know, Data Colada is thinking, all right, this was not just randomly selected observations that were in the extreme. These were observations that already looked suspicious because of their positions in the spreadsheet.
And then it turned out they also had unusually extreme values. And those extreme values all supported the hypothesis. Pretty crazy coincidence, right?
[Kristin] (29:14 - 29:24) Random, like I said, mistake where somebody maybe misentered a few things and then fixed it later. Because if that were the case, you wouldn't expect to see this pattern.
[Regina] (29:24 - 29:42) Exactly. But Data Colada, those guys were not done because then they found something that I think is actually the coolest part about all of this. So, Kristin, before reading this Data Colada post, did you know that an Excel spreadsheet is actually a zip file?
[Kristin] (29:44 - 29:47) Quite a surprise to me, actually, when I read that, but it's really cool.
[Regina] (29:48 - 30:11) It is super cool. You can literally unzip an Excel file and look inside on your browser. And when you look inside, and there are all these smaller files, and those are the files that Excel bundles together to make the spreadsheet that you see on your screen.
And one of them has this wonderful name, CalcChain.xml. Isn't that great?
[Kristin] (30:12 - 30:12) Sounds like calculus.
[Regina] (30:12 - 31:04) I'm going to name my dog that, calcchain.xml. Come here, buddy. Calculation chain is what that stands for. So, CalcChain, just to be clear, it's not a complete history of everything anyone has done.
It's not like Big Brother, recording over a keystroke so you don't have to get paranoid. But it does have information about the order that Excel calculates certain formulas. And so, when someone moves a row or a cell that has a formula, the calculation order can preserve clues about where the formula used to be.
And it gets a little technical, but as the Data Colada people put it, this meant they could try to reconstruct what the spreadsheet might have looked like back in 2010 before the suspected changes.
[Kristin] (31:05 - 31:28) Which is kind of amazing. And actually, there's a little irony here too, Regina, because it reminds me of them telling the participants to throw out their math puzzles, and they think, therefore, that nobody can catch them cheating. It's kind of the same thing here, right?
Whoever mucked with the data probably didn't realize that there was some kind of record behind the scenes. So, I think it's kind of ironic. It sort of mirrors the math puzzles.
[Regina] (31:29 - 32:36) Oh my god, I love that. I didn't think about that, Kristin. That's brilliant.
I like the way your mind works. Yes, it was exactly the same thing. It was coming back to bite them in their butt.
Okay. So, I was thinking of it like footprints or fingerprints, but maybe it's just desserts. Okay.
So, Data Colada examined that CalcChain.xml information for those seven out-of-order observations. And what did they find? The calculation chain suggested that all but one of them had originally been in different experimental groups.
So, for example, one observation appeared in the sign-at-the-bottom group looked like it had originally been in the no-signature group. And another observation was in the sign-at-the-top group looked like it had originally been in the sign-at-the-bottom group. So, they looked like they were moved around between groups, not the group that they were actually assigned to.
And these were exactly the kinds of changes that would make the result look more favorable to the hypothesis.
[Kristin] (32:36 - 33:24) So, this is very suspicious because if you're moving rows of data around to a new group, again, it could have been a coding error at one point. But with so many of them in exactly the direction that favors the hypothesis, this looks very suspicious. It kind of looks like somebody said, hey, this observation has a high amount of dishonesty, so I'm going to move them to the group I want them in.
You can't do that, obviously. We've got observations that are out of order. They're unusually extreme in precisely the direction that supports the hypothesis.
And add to that that Excel's calculation chain suggests that some of them were actually moved. And we're guessing that they were moved because of the out-of-the-order IDs, too. But we can actually prove that the whole row was moved as opposed to the ID number changed.
That really starts to paint a pretty serious picture.
[Regina] (33:24 - 34:54) Yeah, this is pretty damning here. Okay. But there was something else that I wanted to know, and I had access to the data, Kristin.
So, Data Colada showed the suspicious observation seemed to have been moved between groups, right? But I wanted to know how much did those suspected changes, how much did they actually matter to the results, the results that the researchers published? So, I went in and did a kind of sensitivity analysis.
So, first, I just reanalyzed the published data set as it was using the same statistical test that the researchers reported in their paper just to make sure I could reproduce their findings, right? And I could, so that's good. Then I went in and tried to reverse those suspicious changes, like roll them back.
So, using those clues from Data Colada, I put the observations back in the experimental groups where they look like they had been originally, and I removed one of the duplicated observations, and then I re-ran the same statistical test. So, this is not necessarily the actual original data set, right? I'm not saying that.
It's just a hypothetical reconstruction. But I wanted to know what might the researchers have seen before those suspicious changes were made. And it got a little interesting.
So, for all three outcomes, the overall statistical test was still significant. And the differences between signing at the top and signing at the bottom, still significant.
[Kristin] (34:54 - 35:14) So, even without those suspicious changes, the main finding was still there. Well, that's interesting because why would somebody muck with the data if they already had a significant result? Of course, we don't know, Regina, if there was other changes made in the data that were not traceable.
So, this may have been one of many.
[Regina] (35:15 - 36:27) Right, exactly. But that was the thing. I wanted to see, like, okay, what's the motivation here if these were the only changes?
But then I went back to the paper, and I read it a little bit more carefully at the specific comparisons that the researchers had reported. And I realized I'd been overlooking something, basically. So, remember, there were not just two groups.
There was the signing at the top, the signing at the bottom, but there was also that third group, the control group, the no signature at all. And remember what you noticed earlier, Kristin? The people in that no signature group had lower rates and lower magnitudes of cheating than the people assigned at the bottom.
So, that was great sleuthing. That turns out to be important. Because in the published data, signing at the top wasn't just better than signing at the bottom.
It was statistically significantly better than both of the other options. But in this reconstructed data, I analyzed that changed. Signing at the top was still significantly better than signing at the bottom.
But for two out of three outcomes, signing at the top was no longer significantly better than having no signature at all.
[Kristin] (36:27 - 36:35) Oh, wow. I mean, signing at the top was still better than signing at the bottom. And isn't that the main hypothesis they were testing?
[Regina] (36:35 - 37:19) Yes, absolutely. That's still an interesting result. But think about what the researchers actually concluded, right?
They summarized it in that one sentence. They wrote, the signing before reporting promoted honesty, whereas signing afterward was the same as not signing at all. That's a great story, right?
You just have to move that honesty pledge up to the form and people are going to be more honest and that's all you need to do. But with this reconstructed data, they would have had to tell a slightly different story. It would be something more like signing at the top was better than signing at the bottom.
But we couldn't say that it was any better than not signing at all. That's just a little less compelling.
[Kristin] (37:20 - 37:34) Right. Because then the conclusion might be, don't bother changing your forms to put a signature on at all. So there may have been some motivation, you're saying, Regina, for improving, tweaking the results a bit.
[Regina] (37:34 - 37:42) Right. You know, because sometimes it's really a good story that sells something. You get noticed in the press, people pick this up, everyone likes this.
[Kristin] (37:43 - 37:48) Right. You usually are hoping to beat the control group. What you're saying is they didn't actually beat the control group here.
[Regina] (37:48 - 38:07) They did not actually beat the control group. OK, Kristin, let's get back to the saga. 2023 now.
Data Colada has published its findings. We just talked about all of those here. And everything about the controversy, everything about the scandal is now out in the open.
[Kristin] (38:08 - 38:41) Right. And at this point, Gino is on administrative leave. Papers are being retracted.
A lot of accusations are making headlines in The New York Times because it's just too good of a story. Honesty researchers being dishonest. So made some major media.
Yeah. But then, of course, what does Gino do? She decides to fight back.
So in August of 2023, she files a lawsuit against Harvard, the Harvard Business School dean and the three Data Colada researchers. And she was seeking twenty five million dollars in damages from the Data Colada researchers.
[Regina] (38:42 - 38:46) And that's a lot of money. Yeah.
[Kristin] (38:48 - 39:00) It's really uncommon, Regina, for data sleuths to be sued. I mean, maybe we get maligned, maybe we get accused of being too mean. But suing is a whole other level.
[Regina] (39:00 - 39:04) Yeah, they don't they don't send you an email asking for twenty five million dollars.
[Kristin] (39:05 - 39:14) That's right. And the lawsuit includes allegations of defamation and against Harvard. She was also alleging gender discrimination.
[Regina] (39:15 - 39:44) OK, but this is where I get more interesting for people like us, Kristin, that care about the data more than the legal twists and turns, because March 2024, Harvard investigative report. Remember, it's like thirteen hundred pages that become public as part of this whole litigation. Now everyone can read it.
And that report talks about something that Data Colada did not have access to when they did their original sleuthing.
[Kristin] (39:44 - 39:51) Yeah, they got an earlier version of the data set. And Regina, I really want to get my hands on this, but we don't have access to it.
[Regina] (39:52 - 40:36) Not yet. Not yet, at least. So remember, Data Colada had been working backwards from clues in the published spreadsheet and trying to figure out where some observations might have originally been.
And now Harvard investigators had another data set to compare against the published version. So the earlier file came from Gino's former lab manager at the University of North Carolina, where the experiment had been done and it did not match the data set that had been published. Harvard brought in data forensic specialists to examine the differences between the files and just analyze the heck out of everything.
Wouldn't that be a fun job?
[Kristin] (40:37 - 40:45) I know. Data forensic specialist. That's almost as cool of a title as distinguished statistician, though I still I think I want distinguished statistician, Regina.
[Regina] (40:47 - 40:49) Why not have it all? Let's do both.
[Kristin] (40:49 - 40:54) Distinguished data forensic specialist. That's the job I want.
[Regina] (40:54 - 40:57) Card carrying. Distinguished.
[Kristin] (40:57 - 41:03) Card carrying. Establishment distinguished data forensic specialist. Yes.
That's a mouthful, though.
[Regina] (41:04 - 41:14) I'm making you business cards.
[Kristin] And podcaster.
[Regina] And podcaster. We're going to put all of that on your conference badge, you know, when they have your title.
[Kristin] (41:14 - 41:15) Oh, yeah. We need to do that.
[Regina] (41:16 - 41:45) Okay. So forensic specialist, which we want to be. Let's get into what they found.
But first, I really need to mention, I think, that Gino disputes Harvard's interpretation of the evidence. So she and Lessig have argued that the differences between these two data sets could reflect legitimate corrections, right, rather than deliberate manipulation. And they have actually challenged which data set should be treated as the original.
[Kristin] (41:46 - 42:07) Yeah. This is all reminding me of the episode with Kate Laskowski. Data management.
Good data management and version control. Because clearly, at the very least, they were guilty of not having good version control. Regina, we're going to get into those arguments, what they are arguing, more fully in our next episode.
But yes, we want to give some space for their rebuttals.
[Regina] (42:08 - 42:27) Right. Right. Okay.
So now let's look at what Harvard's forensic investigators reported in their big report. And here is what I really find so, I don't know, gratifying, satisfying about the results. Because remember those observations that Data Colada had flagged as suspicious?
[Kristin] (42:27 - 42:34) Right. The ones that were out of order or duplicated with the unusually extreme values in just the direction needed.
[Regina] (42:34 - 43:01) That's the ones. Okay. Well, Harvard's forensic team, they, like I said, compared the earlier dataset with the published version.
And remember we had 101 rows, 101 observations in the published dataset. They identified 11 that did not match the earlier file. And guess what?
All of the observations that the Data Colada folks had flagged were among those discrepancies.
[Kristin] (43:01 - 43:19) Wow. So the Data Colada team had independently spotted the very observations that turned out to be different between the two versions of the data, meaning their statistical sleuthing was pretty good. Although I guess they didn't catch all of them.
[Regina] They did not catch all of them.
[Kristin] I'm not criticizing. I'm just saying it's hard to catch all of them, right? People do cover their tracks.
[Regina] (43:21 - 43:48) Especially because they started with just this Excel spreadsheet that looked like it was slightly oddly sorted and had some, you know, weird observations and, you know, a zip file. And now an investigation that used records that they didn't even have access to, they were identifying discrepancies involving those same observations. I think that they just, I don't know, they need like a medal for this, for their detective work.
[Kristin] (43:49 - 43:50) They nailed it is what you're saying.
[Regina] (43:51 - 44:26) They did. Okay. But there was more in the forensic report.
So the report said that 52 percent of the reported responses in the spreadsheet contained entries that had been, quote, modified without apparent cause. That is a lot. And the differences were not limited to things like participant ID or demographic information, right?
They also involved those outcome variables that the researchers were studying, like things like the travel expense claims.
[Kristin] (44:26 - 44:34) Ah, like those unusually high travel expenses in the sign at the bottom group? They called those out specifically.
[Regina] (44:35 - 44:35) Exactly.
[Kristin] (44:36 - 44:49) So, Regina, you mentioned 52 percent of the responses had changes. I want a little more information there. Do we know, is that just like half of the rows of data had at least one change per row, or do we know?
[Regina] (44:49 - 45:03) Yeah, that was my question, too. They did not say in the report, they just said 52 percent of the reported responses, right? What's the denominator is what you're asking.
So I don't know. I think we need to wait to get the data set ourselves and then we can dig into it.
[Kristin] (45:03 - 45:42) Well, I mean, just to speculate a little here, at the very like most conservative, most favorable to Gino, that might mean something like half of the rows had at least one data point changed. And you could imagine them trying to explain that as somebody went through and audited the data and like corrected any data entry errors. So you could imagine one person entering all the data and then having somebody go back and check all of the entries to make sure they're correct.
I would be surprised if there was an error in half of the rows of data. That would mean that the first person who entered the data did a really, really sloppy job. That would be unexpected.
It's not that plausible.
[Regina] (45:42 - 46:14) Yeah, it's really not clear. It's just pointing to the fact that there were a lot of responses that had been modified without apparent cause. So Harvard's investigators also went back to the different versions of the data and they reanalyzed everything and they reported that the published data set produced results that were generally much more favorable to the researchers' hypotheses than the earlier version of the data set.
[Kristin] (46:14 - 46:21) Which adds to our suspicions because if this was just somebody correcting sloppiness, you'd expect the direction to be random.
[Regina] (46:21 - 46:57) And Kristin, I think there's something really important here as we're going through the whole saga. So finding the two data sets differ does not by itself establish why they differ or who was responsible, right? So that is what is really in dispute right now.
In the legal battles, yeah. But it does mean that the history of all of these changes really matters enormously. And we'll put a link to Harvard's forensic report in the show notes for everyone who wants to read this 1300 pages while they're on the beach.
[Kristin] (46:58 - 47:06) Light bedtime reading, Regina, yes. Okay, so Harvard's report becomes public March 2024. What happened after that?
[Regina] (47:06 - 47:17) Yeah, the legal twists and turns just keep on coming. So September 2024, a federal judge dismisses the claims against the Data Colada researchers.
[Kristin] (47:18 - 47:20) Which must have been a huge relief for them.
[Regina] (47:23 - 47:44) I think it's a good sign your career as a data detective still has a future, right? My triumph in the end. So the judge at the time also dismisses several claims against Harvard, but not the entire lawsuit.
And then in May 2025, Harvard revokes Gino's tenure.
[Kristin] (47:44 - 47:55) Which is kind of extraordinary. This was reportedly the first time that Harvard had revoked a professor's tenure, at least in its modern history of the concept of tenure since like the 1940s.
[Regina] (47:56 - 48:39) I know, it's crazy, right? And just about data, too. Okay, in August 2025, there was another twist.
Harvard files counterclaims against Gino. So Harvard alleges that she created or altered a dataset, changed its metadata to make it appear older, and used that file to challenge the university's investigation. So, of course, Gino disputes the allegations.
And then the litigation, it continues through all of 2025 and 2026. And then a trial was scheduled for December 2026.
[Kristin] (48:40 - 48:43) And we're recording this in October. So that's coming up, but hasn't happened yet.
[Regina] (48:44 - 49:03) Not anymore, because I went digging. Our latest update. Yeah, it was another twist.
Our latest update was on October 6th. Just recently, the judge postponed the December trial. There are still several motions pending, and there's not a new trial date yet.
[Kristin] (49:03 - 49:05) Oh, so it's a cliffhanger for next season. Good.
[Regina] (49:08 - 49:28) It is. And of course, Gino continues to deny committee and research misconduct. And like we said, Lessig has nine podcast episodes and a website and is defending her and challenging Harvard's conclusions.
And that all is actually pretty interesting as well.
[Kristin] (49:28 - 49:55) Yeah, I mean, he is really doubling down on this. And we will explore some of those arguments in our next episode. We're going to look at another of Gino's disputed studies.
That's one about creativity and dishonesty being linked. And we'll look at some of Lessig's arguments supporting Gino and we'll dissect those. But for now, I think it's time to wrap this episode up, Regina, and rate the strength of evidence for our claim.
What was our claim for today?
[Regina] (49:55 - 50:01) Our claim was Excel spreadsheets are neither sexy nor reliable data management tools.
[Kristin] (50:03 - 51:24) Ah, and how do we rate strength of evidence for claims in this podcast on our 1 to 5 highly scientific smooch rating scale, where 1 means little to no evidence for the claim and 5 means strong evidence for the claim. As with a few previous episodes, Regina, this claim is deviating a little bit from the normal health claims that we would look at. And so we don't necessarily have great amounts of empirical data on this subject.
But this one is an easy one for me to rate because Excel is bad, bad, bad. You can store data in it and that's it. And so this is based on personal experience and anecdote more than empirical data.
But I'm going five smooches on this one. I want to just allude, though, to a previous episode we did, Regina, when we had a claim that was a little squishy like this. In the Control Z episode, we mentioned that we would like to see some empirical data on whether or not retracting your own papers is good for your career.
And somebody pointed out on social media that there actually has been such a paper. There was an empirical paper that looked at retractions, and they found that actually a retracted paper did reduce citation counts for that researcher in the future. But if it was a self-retraction, most of the bad effect was actually mitigated.
So you're better off self-retracting than somebody else finding your mistakes.
[Regina] (51:25 - 51:41) Oh, that is fascinating. I love it. So maybe there is an empirical study out there talking about whether Excel is sexy or not.
Okay, I admit it. I put in sexy in there just because it's been a long time since we've had sex in an episode.
[Kristin] (51:42 - 51:49) I know we are really failing on that, Regina. In all of these retraction papers, we haven't managed to sneak in the sex. We'll work on that.
[Regina] (51:49 - 51:57) I vow, I pledge down here at the bottom of the episode that I will bring in more sex next time.
[Kristin] (51:58 - 52:09) I think we're just talking about sexy topics, though, Regina, because statistical sleuthing, come on, that is so sexy to start with. The math is sexy. We don't need to bring in actual sex into these episodes to make it compelling.
[Regina] (52:09 - 52:15) I'm not sure you would have 100% support on that in the population of people.
[Kristin] (52:16 - 52:17) That's another claim for us to take.
[Regina] (52:17 - 52:23) But I'll go with it, too. All right, my smooch rating on this would be five smooches.
[Kristin] (52:24 - 53:03) Yeah, five-five. You don't manipulate, clean, touch data in Excel ever. Never analyze data in Excel.
No touching data in Excel ever. And now we've scared people, Regina, because CalcChain, everybody's going to be afraid about what record is kept behind the scenes in Excel. So there you go.
Transparency, people, you're better off knowing what record is being kept on you than not knowing.
[Regina] Okay, methodological morals.
[Kristin] Yeah, I've got one, Regina.
How about this? There's cleaning your data, and then there's laundering your data. I think some of the defense in this case might be, oh, we were just cleaning up our data, but to me it looks a lot more like laundering than cleaning.
[Regina] (53:03 - 53:09) Data laundering, is that like a thing? Is that a term, a phrase that's out there?
[Kristin] (53:09 - 53:11) We should trademark it. That's great. Yeah.
[Regina] (53:11 - 53:14) Money laundering, data laundering. I know, data laundering. Oh, I love it. I love it.
[Kristin] (53:15 - 53:17) What's yours, Regina?
[Regina] (53:17 - 53:26) I'm going to go back to record keeping. How about this? Good data record keeping is still sexy.
Version control, baby.
[Kristin] (53:27 - 54:00) Absolutely. Come on, a little version control would have had all the answers, and now it's all kind of a mess because there was no good documentation or version control. And maybe they're hiding behind that as an excuse, possibly, but still.
All right, Regina, this has been fascinating. And again, next time we're going to talk about another of Francesca Gino's papers that has been retracted. We'll get into some data sleuthing and also talk about some of her and Lawrence Lessig's defense that they've offered to see what they're saying and what we think of it.
So looking forward to that one, too.
[Regina] (54:00 - 54:04) That is going to be fascinating. So thank you, Kristin. Thanks, everyone, for listening.