572. Why Is There So Much Fraud in Academia? | Freakonomics Radio

Key Takeaway

Treat any appealing research claim as a hypothesis, not a rule for action. Before changing a policy, purchase, or habit based on one study, spend 10 minutes checking whether the result has been replicated, whether the underlying data or methods are available, and whether the claim sounds implausibly simple. When the stakes are meaningful, test the idea on a small scale yourself and define in advance what evidence would change your mind.

Episode Overview

This episode examines alleged data fraud in high-profile behavioral-science research, focusing on Francesca Gino, Dan Ariely, and the influential claim that signing a form at the top reduces dishonesty. It follows the researchers and data detectives who challenged the findings, while exploring how career incentives, weak transparency, and selective analysis can undermine scientific credibility. The episode also distinguishes failed replication from deliberate fraud and argues for stronger verification practices.

Full Transcript

Transcript of 572. Why Is There So Much Fraud in Academia? | Freakonomics Radio from Freakonomics Radio. Auto-generated from episode audio; may contain minor errors.

Just over a year ago, Francesca Gino was, there's no other way to put it, at the top. She was a superstar, an academic superstar. At least she was at the center of it all. As a prestigious professor at Harvard, with all her lectures and books, her reputation was impeccable. She was synonymous with the highest level of research in organizational behavior. She's simply a giant in the field. The field in which Gino is a giant, where his reputation was impeccable, is called by various names: behavioral science, decision science, or organizational psychology.

According to her website at Harvard Business School, where she is a professor of business administration, Gino's research focuses on why people make the decisions they make at work and how leaders and employees can have more productive, creative, and fulfilling lives. Who wouldn't want that? Gino became a superstar by publishing a large number of research articles in academic journals, as well as several books. Her most recent work is called Rebel Talent: Why it Pays to Break the Rules at Work and in Life. She produced the kind of media-ready research that fits perfectly into the virtuous circle of academic superstars.

A scientific article is amplified by the publisher or university in the mainstream media, which fuels headlines for all companies and institutions eager to exploit the next insight in behavioral science. And this, in turn, generates an even greater appetite for more useful research. The academic capable of consistently producing such work is treated almost like an oracle. There are TED talks to be given, books to be written, and consulting jobs to be landed. Francesca Gino, for example, has given lectures or provided consulting services to Google, Disney, Walmart, the US Air Force, Army and Navy, and many others.

But all that is over for now. In July 2023, Harvard Business School, responding to an analysis by academic whistleblowers, investigated Gino's work and found that she had, quote, intentionally, knowingly, or recklessly committed research misconduct. Gino was suspended without pay. She then sued Harvard and the whistleblowers. These same whistleblowers also produced evidence of what they call data fraud by an even more prominent behavioral scientist, Dan Ariely, from Duke. Ariely has been enjoying the spotlight for many years, ever since her 2008 book, Predictably Irrational: The Hidden Forces That Shape Our Decisions.

Duke University is reportedly finalizing its investigation into Ariely, although this has been ongoing for some time. And when it comes to academic fraud, universities have a habit of downplaying accusations against their superstar professors, for the obvious reason that it reflects badly on them. Meanwhile, Dan Ariely's book continues to serve as the basis for a new NBC crime drama called The Irrational. It stars Jesse L. Martin as a professor who uses behavioral psychology to help solve crimes. Hey, what were you doing back there? Paradoxical persuasion. I accepted his idea excessively to force him to think about it enough and realize it was a terrible idea.

And how did you know he wasn't going to pull the trigger? It works about 95% of the time. And what about the other 5%? There are always exceptions. Lissa. Dan Ariely and Francesca Gino maintained their position that they never fabricated data for their research. None of them agreed to record an interview for this episode, but one of its co-authors did. Certainly, I felt a moral obligation to correct the records. Today on Freakonomics Radio, we'll hear from him and the three data detectives who exposed Gino and Ariely.

So, I would say that I am certain about the falsity of the findings. I have no reasonable doubt. But this is a much bigger story than just two high- profile cases in behavioral science . We will delve into the incentives that generate academic fraud. If you were simply a rational agent acting in the most selfish way possible as a researcher in academia, I think you would cheat. We will hear what is being done to change this and, more importantly, why it matters, because fraud in academic research goes far beyond academia and has consequences for all of us.

The first part of a two-episode series begins now . This is Freakonomics Radio, the podcast that explores the hidden side of everything with its host, Steven Dubner. I rarely do this, but today I'll start by reading a few sentences from Freakonomics, which Steve Levitt and I published in 2005. Cheating, we wrote, may or may not be in human nature, but it is certainly a prominent feature in almost every human endeavor. Cheating is a fundamental economic act : getting more for less. So, on second thought, why shouldn't we expect cheating even among scientific researchers?

Consider this. Today it is believed that Ptolemy, the 2nd-century Greek astronomer , fabricated his observations to fit his theories. And a new study in the journal Nature found that, last year, more than 10,000 research articles were retracted, easily surpassing the previous record. Although many of the recent headlines are about scholars at prestigious American universities, the countries with the highest number of retractions were Saudi Arabia, Pakistan, Russia, and China. Fraud has existed for as long as science has existed. This is mainly due to the fact that human beings do science, and people bring ideas, beliefs, motivations, and reasons for conducting the research they do.

And in some cases, people are so motivated to promote an idea or themselves that they are willing to fraudulently alter the evidence to further that idea or their own interests. This is Brian Nosk, a professor of psychology at the University of Virginia . In 2013, he founded the Center for Open Science, a non- profit organization that seeks to improve the integrity of scientific research. Just to be clear, I asked Nosk where their funding comes from. Our funders include the NIH, NSF , NASA, and DARPA as federal sources.

And then a variety of private sources , such as the John Templeton Foundation, the Arnold Foundation, and many others. And this diverse group of funders, and it is quite diverse, I believe shares the recognition that the substantive issues they are trying to solve will not be effectively resolved if the work itself is not done with credibility. In other words, what's at stake here is significant, much higher than just individual academic researchers trying to advance their careers. If your goal is to improve medicine, transportation, or immigration policy—any area where decisions are based on academic research—you don't want that research to be compromised.

There are specific cases in which a discovery is translated into public policy or some type of activity that ends up harming people, lives, treatments, and solutions. One of the most prominent examples is the Wakefield scandal related to the development of autism and the notion that vaccines could contribute to it, which had an incredibly corrosive impact on public health, people's beliefs about the origins of autism, and the impacts of vaccines, etc. And this is very costly for the world. There is also a local cost in academic research, which is simply a ton of waste.

So even if it doesn't have downstream public consequences , if a false idea is in the literature and other people are trying to build upon it, it's just waste, waste, waste, waste. There's also the idea that, however much universities worry about their students cheating, such as using ChatGPT to write a paper, what kind of example are their professors setting? And there's another big reason why this story is so frustrating, and that has to do with the standards of academic research. The general view, at least the view I have always held, is that academic research exists in a special category.

It is a fact-seeking coalition that operates under a set of rules built around the accurate collection and analysis of data, with the entire process subject to fact-checking and peer review. Good journalism operates under similar rules . The New York Times has a long-standing mission statement that I've always liked. He says he must report the news impartially, without fear or favoritism, regardless of party, sect, or interests involved . I have always believed that this mission also applies to academic research, that it should be not only accurate, but free from personal or financial interests.

These research articles are not being written by any political official, management consultant, or stock analyst. These are being written by someone so dedicated to their field of research that they went through the hell of obtaining a doctorate to spend their days doing this research. But the fact that Brian Nosk is so busy with his open science center suggests that my faith in academic research was misplaced. I asked Nosk to explain to me how he went from being a researcher to a new type of referee. Yes, I've always been interested in how to do good science as a matter of principle.

And in doing so, we in the lab were working on developing tools and resources to be more transparent with our work, to try to be more rigorous, to try to create more robust and sensitive research designs. So, I wrote grant applications saying: can we create a repository where people can share their data? You know, this was around 2007. And they were getting polarized reviews, where some reviewers said this would change everything. It would be so helpful if we were more transparent about our work, and others are saying that researchers don't like to share their data.

Why would we do that? And why wouldn't researchers want to share their data? Yes, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance in my career, to get promoted. So, regarding the work I do that leads to publication, I have a very strong feeling that, oh my God, if others now have control over this—over my ideas, my data, my projects, my solutions—then I will harm my career. Wow, I feel so naive, because I've spent a lot of time in this ecosystem for many years now.

What you're saying simply sounds wrong and sounds selfish. And, worst of all, it seems to go against the mission of the scientific objective, which is to discover and disseminate knowledge to the world. And that, in a way, offends me, I must say. Yes. And here's the irony: almost every academic would say that, of course, science should be transparent. It's clear that we're doing research for the public good. Of course, all of this should be shared. Okay, Polyana, here we go. We live in a world, right?

The reality is that there's a reward system, and I need to have a career to do this research. So yes, we can talk about these ideals of transparency, sharing, rigor, and reproducibility, but if they're not part of the reward system, you're asking me to act according to my ideals and not have a career, or to have a career and sacrifice some of those ideals. You can see how these incentives create a problem. If a system has a built-in bias against transparency, not only will there be less transparency, but also more opportunity to cheat.

Nosek and his colleagues decided to approach this by trying to replicate the results of articles that had already been published in academic journals. They called their idea the reproducibility project. And at the end of 2015, we published the results , which consisted of an article with 270 co-authors on 100 replications of findings from three different psychology journals. We were able to successfully replicate slightly less than half of the findings. You didn't hear wrong, Brian Nosek. That's exactly what he said. Slightly less than half of the findings were successfully replicated.

So he has been carrying out large-scale replications ever since. And not just in psychology. A year and a half ago, we published the results of the reproducibility project in cancer biology, following the same type of process, and we found very similar results. Less than half of the findings in preclinical cancer research have been successfully replicated when we've tried to do so. It's important to point out that what Nosek says here may not be as bad as it seems. See what he thinks about this high number of studies that are not replicated.

This does not mean that the original finding is necessarily wrong. We might have made a mistake in the replication process. Successful replication does not mean the interpretation is correct. It's possible that both findings contain a confounding factor, but we were only able to replicate the confounding factor. There may be other explanations as well, legitimate explanations. Therefore, we should not equate a failure in replication with fraud or even misconduct. Still, Nosek's figures suggest that much of the research being done today may not be producing science of lasting value.

And we're talking here about research conducted at the most elite universities and academic journals. If you're a fan of science, of any kind of science, this should concern you. Fraud is the ultimate corrosive element of the scientific system. Because, however much transparency may provide a substitute for trust, one cannot be transparent about everything. So, the ideal model in research is that you can see how the evidence was generated, how it was interpreted , what it really is, and then independent people can examine it. And to the extent that fraud creeps in and the evidence doesn't actually exist, it's not real evidence, the whole edifice of this academic debate and this clash of ideas collapses, because you're confronting ideas that aren't based on anything.

How much do you know about Yookim Bolt's situation? I ca n't remember that name, but perhaps I'll recognize the case if you describe it. Yes, that was the German anesthesiologist who had almost 200 articles retracted, and oh yes. From what I understand, there were people who actually died as a result of this flawed research. So, I'm curious to ask in which academic fields or disciplines you think fraud or negligence is most prominent? We cannot say with certainty where it is most prominent. We can only say that the incentives for this are everywhere.

And some of them get more attention because, for example, Francesca's discoveries are interesting. They are interesting to everyone. So, of course, they're going to get attention for it. While the anesthesiologist's findings are not interesting. They make people sleep until they kill you. Well, yes, I think they make you fall asleep and then kill you. But it seems that his field of study, psychology, and especially social psychology, is a hotbed of, I wouldn't say fraud, but certainly of controversy and overturned findings over the years. Yes, I would say that there is a lot of attention focused on social psychology for two reasons.

One is that it has value in terms of public interest and engagement. This is a way of saying that people care about your discoveries. People care, at least in the sense of "oh, that's interesting to learn," right? But the other reason is that social psychology has taken the trouble to investigate, and I think social psychology has become a focus of this because the real challenges in the social science system that need to be addressed are psychosocial problems. What do you mean by that? I mean, like the reward system, how can people rationalize and use motivated reasoning to arrive at conclusions that are less credible?

Many of these problems are ones that social psychologists think about every day. I asked Nosk why he thinks that presumably honest people can, over time, begin to behave dishonestly. The argument people use to say that this is a bigger problem now is that competition for attention, jobs, and promotions is very high, perhaps higher than ever before . Do you think this has been driven by the reduction in permanent positions at universities? Yes. So there are many more people and far fewer job openings, which is an obvious challenge in a competitive market.

And now there are avenues for public attention that have an even greater impact. Academics, in general, did not think about ways to get rich. They were looking for ways to have time to think about the problems they wanted to analyze. But now they have ways to get rich. These paths have benefited many people, including myself, even though I am not an academic. Thanks to my partnership with Steve Levitt, an economist at the University of Chicago, I've had more opportunities than I ever imagined possible, including this program.

I wonder why there seems to be far less questionable research in economics than in psychology and other fields. When you talk to economists, they will give you various reasons. Economic research is very mathematical and comes with many so- called robustness tests. There is also a tradition of, shall we say, aggressive debate within academic economics. Long before publishing an article, you typically present it to colleagues and veterans at seminars, and they are happy to point out any possible flaws and call you an idiot if you disagree.

I'm not saying this is the best way to conduct the work, but it certainly makes dubious data more costly. Furthermore, economists tend to work with large datasets, much larger than those in other social sciences, and these datasets are often publicly available. Therefore, there is no opportunity for cheating there. This does not mean that there is no controversy in economic research or that results have not been refuted. There are many. And if you associate with economists, as I often do, you'll hear whispers about some researchers suspected of manipulating their data.

But it seems that most researchers in economics are largely honest. And for all honest researchers in any field, there's yet another twist. When you play by the rules, but see that the people who are winning the game are cheating, you feel like a sucker, and nobody likes to feel like a sucker. But it's something bigger than that. If cheaters are winning, it means that those who don't cheat receive smaller rewards, and it means that all their hard work may also be viewed with suspicion. So, what can be done about this problem?

This may require something more invasive than a reproducibility study. This may require questioning the data or research methodology in suspect articles and making public accusations of fraud. And that brings us to the whistleblowers we heard about earlier. They are co-directors of the credibility lab at the University of Pennsylvania and, collectively, maintain a blog called Data Colada. My name is Leif Nelson and I am a professor of business administration at the University of California , Berkeley. Uri Simonsohn. I am a professor of behavioral science at Esade Business School in Barcelona.

Joe Simmons. I am a faculty member at the Wharton School, University of Pennsylvania. Nelson, Simonsohn, and Simmons are, let's say, mid-career academic researchers. They have been working in the field for some time and hold high- level positions within the ecosystem that concerns them. They have all published extensively in leading psychology and decision science journals . Therefore, they approach this not only from within, but with insider knowledge of how academic research works and where it falls short. Therefore, they focus on examining the methodology used in this type of research.

What motivated our entire journey in this methodology was that we used to go to conferences or read articles and not believe them. And we realized that whenever a discovery didn't align with our intuition, we trusted our intuition more than the discovery itself. And that , in a way, thwarted the whole purpose. Like, if you only believe what you already believe, then why bother? This guy, Daryl Bem, published an article with nine studies, eight of which had statistically significant evidence that people have extrasensory perception. And most people were thinking, "What's going on?" As if this couldn't possibly be a genuine discovery.

So the idea was: how do we show people that you can produce evidence of anything very easily? So we thought we'd start with something that's obviously false. We said: okay, something that's really hard to do is make people younger. We've been trying this for a long time. We were never successful. So, let's show that we can do this in a silly way. We decided to show that we can make people younger by having them listen to a Beatles song. The song was "When I'm 64". Correct.

That's right. And the idea is: if we can make anything meaningful, one way to prove it is to say: I'm going to show with statistically significant evidence that people got younger after listening to "When I'm 64". They conducted real laboratory experiments with real research subjects who had real birth dates and played real music for them. "When I'm 64" and two others. A control song, which I believe is called "Kalimba," by Mr. Scruff. And then we had another song that was supposed to go in the opposite direction, but it did n't work, so we simply did n't report it, which was "Hot Potato".

They essentially manipulated and selected their data to produce the absurd result they wanted: that listening to "When I'm 64" reduces your age by a year and a half. It turns out they published the article in Psychological Science, one of the leading journals in the field. Their work was called Psychology of False Positives: The undisclosed flexibility in data collection and analysis allows anything to be presented as significant. They wrote: "These studies were conducted with real participants, employed legitimate statistical analyses, and were truthfully reported." However, they seem to support hypotheses that are improbable or necessarily false.

So we were surprised when the article was accepted, and the immediate consequences were shocking; Many people identified with that, but there were also many who were not happy at all, asking why we were tarnishing the reputation of the area. But Simmons, Nelson, and Simon felt that their area already had a bad reputation. Yes, we thought it was very bad. We started our blog at the end of 2013. We decided we wanted to have a blog because we thought it would be fun to write shorter things than an academic article and not have to wait two and a half years for the review process.

With that in mind, all we needed was to come up with a name, and we wanted something related to what we do. Perhaps the data aspect was important, but it definitely didn't come across as overly serious. So we tried a few options and, among them, Data Colada was one we obviously stopped at. It had this funny aspect, since Yuri is Chilean, and when he suggested the name, he thought it rhymed, which still amuses me and Joe, because for him it's "date glued together ". The Data Colada team, or Data Colada, or perhaps Data Colada, wasn't just looking for cases of blatant fraud.

They were concerned, as was Brian Nosek, that the pressure to publish interesting results could produce unreliable findings, even if the researcher had followed the rules most of the time. Consider, for example, a practice they came to call P-hacking, with the P standing for probability. This is Leif Nelson. The classic forms of what we characterize as P-hacking are not exactly mistakes. These are decisions that are accidentally biased. It's when you measure several things, but only report the one you like the most. Either you conduct a study with three treatments, condition A, condition B, and condition C, but in the end, you discard condition B and don't even mention it.

You're simply comparing A to C. Remember that this is one of the things the folks at Data Colada did in their joke article about "When I'm 64". They simply left out the "hot potato". And then there are things that are slightly statistical, but in a very unassuming way. Well, we collected this data, but it's a bit biased. They contain some outliers. And you say: should we eliminate these values, or should we use the winsorization method, which basically reduces the extreme values ​​to a defined limit? Or you could process them using an algorithm and say: let's transform them with a logarithm or a square root.

And all of these are justifiable decisions. They are not absurd at all. The problem arises when you have to choose to report one variable or another, and one variable supports your hypothesis while the other makes it less convincing. You end up reporting the one that seems best, either out of self-interest or, honestly, because you think: I'm not sure which one is best, but my hypothesis says it should be this one, so it's probably the best measure. And here is Simonson. There are several approaches. Some of the methods we use involve only statistics, concluding that something is statistically impossible.

Another is identifying associations in the data—or the lack thereof—that are not mathematical properties, but that anyone familiar with data would recognize as incorrect. Imagine you have weight and height data, and when you correlate them, you find a zero correlation. This can't be right. Taller people are heavier. So, if you found a zero or negative correlation, you would think that perhaps these weight measurements are not real. Another approach is to look for rounding errors or suspicious precision. You see rounded values ​​where they shouldn't be, or a lack of rounding where it should be.

For example, in one case we worked on, data was collected by asking people how much they would pay for a t-shirt. And the most curious thing is that there was no rounding. People were equally likely to say 7, 8, or 10 dollars. But if you've ever collected data like this, you know that people round off. People say 10 or 20. They don't say 17. There's another category we could call convenient errors. Here 's Nelson again. These can be simple things like a typo, where someone is writing a report and the averages are 5.1 and 5.12, but the person writes 51.2.

And you think, "Wow, what a huge effect, isn't it?" And nobody corrects it because it's a huge effect in the direction they expected. And literally, a typo can end up being printed, and that's before we get to something like fraud, such as the active fabrication or manipulation of data. Do you think the first set of conditions you described has a high probability of leading to fraud? I mean, how slippery is that slope? The type of person willing to do these things would be willing to go ahead and commit fraud, especially if they've gotten away with it for a while?

Well, Stephen, you were asking a pretty tough question, one that I 'm not particularly well- prepared to answer. If you had asked me this 5 years ago, I think I would have been more refined in my answer and would have said: "No, this is not a slippery slope problem ." There is a slippery slope between "I collect five measurements and report one" versus "I collect 10 measurements and report one". This is a slippery slope. But inventing data seems qualitatively different. And I still largely hold that view.

But there have been enough reports that other people, whistleblower types, have presented to us, that sound much more like someone saying, "Yeah, you know, at the beginning you do that thing of discarding some measures, discarding a condition, or removing outliers, and then you take participant 35 and change his response from seven to nine." You think: "Wow, that last part doesn't sound the same." But perhaps there's a psychology to it, of which it seems to be an extension. So, these three self-proclaimed data detectives investigated the work done by their peers and published their findings publicly.

The first blog post we published was about identifying fraudulent data, in an article published 10 years ago. And this was discovered because Yuri had created a graph for a completely different article, where he was mining data from multiple published studies just to make a graph. And I looked at his figure, along with the data from that other research group, and said, "That looks unusual." I want to read this article. So I read the article and then looked at that dataset. And that one had collected data on a 9-point interval scale.

So , people could answer from 1, 2, 3 up to nine. And there were numbers in the dataset that were things like -1.7. And then you say, "Ah, okay. We're done. Nothing complicated." As soon as you open the dataset, you can close it and say, "It's broken ." The article Nelson was describing had been published by four Taiwanese researchers in 2012 in the journal Judgment and Decision Making. Following an investigation by Data Colada, the article was retracted. Although, as far as we can tell, the researchers were not sanctioned or punished.

So, how much fraud is there? I asked Yuri Simonson. I would estimate the fraud rate to be on the order of, say, 5% of the articles. What is the difference between, say, high-level academic journals and those of intermediate or lower level? Is fraud more likely to be prominent at the high or low level? I don't read the really low-level ones, so I don't know. Sometimes I read about it, but if I find fraud there, I ignore it because the cost of investigating a fraud case is so high that it's simply not worth it.

If it's an article that has, you know, seven citations after 3 years and is published in a journal that nobody knows, I simply let it pass. And I'm sure other people do that too. Simonson, Nelson, and Simmons continued their investigative work alongside their regular research and teaching responsibilities. In psychology and behavioral science circles , Data Colada became well-known and somewhat feared, but they didn't have much reach beyond those circles. That changed a few years ago. They published a post titled " Evidence of Fraud in an Influential Field Experiment on Dishonesty." The fraud they claimed to have identified was in an article published years earlier in a top-tier journal known as PNAS, or Proceedings of the National Academy of Sciences.

The article was titled "Signing at the beginning makes ethics evident and reduces dishonest self-reporting compared to signing at the end." Let's explain this in simple terms. The article stated that if you are asked to sign a form at the top before filling in the information, you are more likely to be honest than if you sign at the bottom. You can see how that would work, right? There are four things you might want to know about this article. Uma, the central discovery, proved to be extraordinarily popular.

Many companies and institutions have started putting the signature line at the top of tax returns, insurance forms, and things like that . Number two, the article was edited for PNAS by Danny Kahneman, perhaps the best-known living psychologist and one of the most respected. Number three, two of the five co-authors of the article were among the best- known people in this field, Dan Ariely and Francesca Gino. Fourth, there was already evidence that something was amiss with the original article, because its authors had published a second article stating that their original findings had not been replicated.

The failure to replicate, as we heard earlier , does not necessarily mean fraud . But now, investigators at Data Colada claim to have evidence that, yes , there was fraud in the original article. This original article was written by Dan Ariely, Francesca Gino, along with Nina Mazar, Lisa Shu, and this man. I am Max Bazerman and I am a professor at Harvard Business School. After the intermission, Max Bazerman guides us through the collaboration that triggered a crisis . Furthermore, Freakonomics Radio had an excellent 2023. You are one of the more than 2 million people who listen every month.

We'll reach 3 million by 2024, okay? If you like the work we do, please tell people about it. This is the best way to support the podcasts you love. Thank you in advance . We'll be right back. Max Bazerman, a professor of business administration at Harvard Business School, is considered a veteran in the field of behavioral science. For decades, he has published highly regarded research articles and books , and is known as a wise and caring mentor to younger academics. This last part, it seems, is the best explanation for how Bazerman ended up co-authoring what would become a very, very, very problematic research paper .

This is the article about " signing at the top" that we heard before the break. The article published in PNAS in 2012 stated that you are more likely to be honest if you sign a form at the top before filling in the information than if you sign at the end. This article actually began as two separate research projects. Here is Bazerman . So, in 2011, Lisa Shu, Francesca Gino, and I had a working paper that was rejected by some journals, which basically aimed to show that if you sign a document before filling it out, you are more likely to tell the truth; Our studies were conducted in a laboratory.

Lisa Shu was a doctoral student at the time. Bazerman was her advisor, one of the chairpersons of her dissertation committee. As for Francesca Gino, Francesca began attending Harvard Business School as a doctoral student from Italy. In 2004, she was attending my doctoral seminar and we began to interact regularly, and eventually I was part of her dissertation committee and played a fairly active role in her supervision. Bazerman liked Gino very much. In fact, almost everyone liked Gino and admired his intellect and work ethic. We even got to the point where our two families made an offer to a construction company for a project to have houses connected to each other.

Their signature at the top of the paper was based on data from two laboratory experiments at the University of North Carolina at Chapel Hill, where Gino taught before coming to Harvard. In these experiments, research participants attempted to solve various puzzles with a financial reward for each correct answer. And then, they filled out a form to inform the researcher how much money they had earned. Some of the survey participants were asked to sign an honesty pledge at the top of the form, others at the bottom. In the experiment, those who signed at the top were more truthful about their earnings.

At least that's what the data said. They were brought to our trio by Francesca, and the exact role of Francesca and the lab manager remains somewhat unclear, but it's certainly safe to say that Lisa Shu and I had little to do with the data collection. This goes back to a time when Francesca was developing an excellent career, but she wasn't the superstar she became, and I was a senior person and probably doing the minimum amount of work, working more after the document was finalized. Do you think that most people, say , the average American, might have a reasonable opinion about university life and academic research, which is perhaps not the average person?

Perhaps the average person wouldn't have such a positive opinion, but for someone who can read an article based on an academic study and say, "Oh, that's interesting. I'll save that as useful and probably true information." How surprised do you think that person would be to learn that a senior colleague like you, who co-authors many articles with junior colleagues, doesn't personally interact with the original data? How surprising do you think most people would find that? So, I would n't say that I don't interact at all .

Hmm, I would certainly read the results section very carefully, but I would read it with the intention of seeing if there were any errors along the way. But how can you know if there's a mistake if you're not, you know, this whole issue reminds me a bit of being, say, a chef in a restaurant and being given the ingredients to cook with, but not being allowed to examine them. So, I don't know if they're rancid, fresh, or even fake. So, I like this example. So, instead of being the chef of one restaurant, let's imagine you own 12 different restaurants and have a head chef at each one.

And this chef will be the one I consider the most experienced of my younger colleagues on the project. And over time, I've come to trust that they will do an excellent job overseeing the ingredients that make up the research process. And in doing that, there are other things I can accomplish. I can work, you know, making sure we have the funds available. I could, you know, work on any specific administrative issues that might arise . I could work with more young academics because I have more time available.

So, there are many benefits to the efficiency of relying on the project's assistant professor or the chef of a specific restaurant, so that I don't have to examine the specific ways in which the sausage is made. Now, remember that this original article was rejected by several journals. Bazerman says they received feedback suggesting that their argument about signing at the top would be more credible if, in addition to the lab results Francesca Gino provided, they also had real-world results. This is what researchers would call a field experiment rather than a laboratory experiment.

Fortunately, another researcher, a friend of Gino's, no less, apparently had some good field results. Collectively , we heard that Dan Ariely was presenting a very similar result based on a field experiment involving an insurance company, and Francesca contacted Dan, and we basically joined forces to bring the three studies together into a single article. Ariely's data included the number of miles that customers of that insurance company reported driving in a year. If you think about how insurance works, a customer might have an incentive to underreport their mileage in the hopes of reducing their insurance bill.

Ariely's data showed that customers asked to sign the mileage statement at the top reported driving more miles than customers asked to sign at the bottom. Again, suggesting that signing at the top makes people more honest. Max Bazerman has now received the first draft of a new paper that combines the studies of Ariely and Gino. I am reading about the insurance company's field experiment for the first time right now. And as I read, I have some questions. I wasn't even remotely thinking of fraud as a problem.

I just thought something was wrong. And what I saw as wrong was that, um, we were reporting that the average driver in the database had driven between 24,000 and 27,000 miles per year. And I just looked at it and said, " That looks strange." It seemed strange to Bazerman because the average American drives only about 13,000 miles per year. So I asked a few questions about it and Arieli, who was the person responsible for this data, whose origin is not entirely clear, sent a very quick email saying that the mileage was correct.

And I continued, saying: well, we need to clarify what's going on here. It seems strange that people would have driven so many miles, especially when we're talking about tens of thousands of drivers. And eventually, Arieli replies that the drivers are, uh, elderly in Florida. It seems they should drive even less than 24,000 miles, then. Exactly what I thought. So my questions continue and I'm not getting very good answers, and this has literally been going on for months, and I'm seriously thinking about removing my name from the article.

At the time, Lisa Shu was a doctoral student in the job market and was presenting this work. And how worried were you about jeopardizing her prospects? I was very worried that if I left the article out, it would seem like there was something suspicious about Lisa's presentation. So I kept asking questions, but I didn't give up. And at the beginning of 2012, I was attending a conference and I arrived there, and in the main hall I met Lisa, who is my advisee, my friend, my co-author, someone I like very much, and she was with Nina Mazar, whom I had never met before.

Then Lisa introduces us, and I believe I expressed my dissatisfaction with the lack of clarity regarding this mileage issue. Nina Mazar is a collaborator of Ariel in the insurance study. Correct. So this is a bit confusing. Okay. So, when we asked Ariel to join forces, he said that was fine, but that Nina would also be part of the project. So I always thought she was part of the study of insurance. Later, Nina claimed that she had no more connection to the study of insurance than I did.

She only connected with him when this five-author article was put together. But, in any case, at this conference, she calmed him down to a certain extent. Yes, exactly. So, she basically, in a pleasant and open way, opened the database on her computer and I said, "So, what's going on?" She said, "I think what's happening is that we don't know if the period between time 1 and time 2 for assessing the number of miles traveled was one year." We know when time 2 was collected, but it may have been more than a year since time 1 was collected.

And in my mind, what's clear is that this makes our study noisier. But as long as a real experiment has been carried out, this is actually very good news. All we need to do is correct the presentation in the article, which we have done. The article was submitted, it was published. I've developed the belief that this effect is real, and people love this result. And, from a theoretical point of view, it's a surprisingly simple idea. From a practical point of view, it's simply perfect. It's so simple that organizations can implement it easily.

And who implemented it? Many people have implemented it. You know, I think Lemonade Insurance, on Ariel's advice, implemented it. Lemonade Insurance, by the way, didn't just follow Ariel's advice. They hired him as their director of behavioral sciences. And many government agencies have implemented it, including the U.S. government . In fact, the first sentence of the article that Baserman and the others published in 2012 says the following. The annual tax gap between actual and declared taxes in the United States is approximately $345 billion. Now, imagine working for the IRS or any tax agency in the world and discovering that these brilliant academic researchers from Harvard and Duke have found that if you simply ask people to sign their tax forms at the top instead of the bottom, millions , maybe billions of extra dollars will suddenly flow into your pockets.

And in 2016, I received an email that was actually fundamental to the entire evolution of what happened afterward. The email is from Stuart Baserman, but he spelled his name with an S instead of a Z. And he's basically a low-key guy who was working at an online insurance startup. And his wife , Sue, encouraged him to email me , as he was working on the problem of "How do we get people to tell the truth online?". And he had read the 2012 article "Sign First" and said, "Looks like maybe I 'm related to a guy who knows how to get people to tell the truth." Then Stewie sends me an email and we develop a very good relationship.

We discovered that he is my fifth cousin, according to 23andMe . And I also developed a consulting relationship with Slice Insurance, the company he was developing. Now, excuse the blunt question, but in retrospect, does this seem like a conflict of interest? Work for my cousin? Well, being a consultant for an insurance company based on the findings of an article that turned out to be fraudulent and which you co-authored. I didn't know it was fraudulent in 2016. And I even believed it at the time. Yes. Are you still a consultant at Slice?

No. Did you leave because the discovery was fraudulent? No. No. I have a great relationship with Slice. Okay. So, going back to 2016, Max Baserman wanted to help his new cousin figure out if signing at the top would be as effective in an online environment as it seemed to work with paper documents. So, Baserman and some younger colleagues decided to test this question. And how was Baserman feeling at the time about the original discovery of signing at the top? We know it works. We know the effects are significant.

We know the world is intrigued by this. It looks perfect. Were you worried about a placebo effect? In other words, if enough people have heard through the media about this sign- first phenomenon, if they encounter a form where they are asked to sign first, they will know that they are under some kind of scrutiny perhaps and, therefore, are more likely to be honest because they know this. This is an excellent methodological critique . Hmm, so what you just said makes logical sense . I think that, at the time we were doing these online studies , and there are many of them.

I don't think there was widespread public awareness of the effect of signing first. You know, Arieli, Gino, and I have spoken with many groups of executives, but I wouldn't say it was a well-known social phenomenon, but I could be wrong. Therefore, your methodological critique could be viable. But, in any case, signing up first, with or without a placebo, doesn't work online. We had nothing . How surprised were you? Very. And I kind of said, well, let's take a look at what we've done. Let's see how we might have messed up the project.

Let's try again. So, we make some changes. We're doing it for the second time. We're making some changes, we're doing it for the third time. Still without effect. No effect. No effect. And remember, the 2012 article not only has effects, it has effects in three different studies that are all statistically significant, and the effects are large. So, the project clearly falters sometime between the third and fifth replication failures. It's transforming from "how to get people to tell the truth online" into a massive replication failure of a rather visible academic effect.

And then, after failing six times, we decided, well, let's go back and do a large-scale replication of one of the original laboratory studies. Baserman and the collaborators he brought in to perform the online replication work met with the authors of the original paper, including Dan Arieli and Francesca Gino, and decided to replicate one of the laboratory studies from that paper, but using more than 10 times the number of research subjects as the original. A characteristic of many academic studies, especially in a field like psychology, is that they often use a small group of research subjects to conduct this type of experiment.

Often only students from their own university. A small research group is cheaper and faster. While speed is good when the goal is to produce a large number of publishable studies, small sample sets are more likely to generate skewed results. So now, with a larger sample and much more rigor, they don't get any effect. Signing at the top doesn't seem to accomplish much. I certainly felt a moral obligation to correct the record. And it's not just a moral obligation, but as you told us, there are institutions, government agencies, and companies that are using this research.

Did you feel, I mean, I don't want to put words in your mouth, but was it a feeling of, um, guilt or panic or fear, something like that ? I would certainly feel some sense of, perhaps, guilt is the right word, for having my name in an article that people are using, when I no longer think they should be using it. But, you know, I didn't think I was doing anything wrong. And honestly , I'm not thinking about fraud right now . I, I just don't know what's going on.

I'm thinking of clearing the registry. In 2020, Baserman, along with all the original authors and their two most recent collaborators, published a companion article in PNAS, the same journal where the original article was published in 2012. This article was titled "Signing at the beginning versus at the end does not diminish dishonesty." From the outside, you might think this isn't a very courageous position to take. You're simply publishing a new article saying that the article you published 8 years ago, the one that received so much attention and boosted so many careers, actually didn't work.

On the other hand, one could say that this is how science should work. You have a hypothesis. You conduct experiments to test your hypothesis. You gather and analyze the data and present your findings. If new information emerges and invalidates your discovery, well , that's what needs to happen to correct the scientific record. But it's worth noting what the original authors didn't do. They did not retract the original article, nor did the journal retract it. At least not up to this point. So, from the outside, this seemed like a story of science that may have been conducted carelessly, but not a story of fraud.

And at least some of the original authors hadn't given up on the original discovery . Arieli and Nina Mazar continued to argue, well, it works sometimes , it doesn't work other times. And we need to do more studies to find out when it works and when it doesn't . And my attitude, I'm not going to speak for the other co-authors, um, was basically to say that we have more than enough evidence to conclude that we should tell the world that we don't trust that this effect works.

So, we didn't rectify it and, uh, life goes on. And in June 2021, I believe , I received an email from one of the Data Colada team members saying, "Max, can the three of us meet with you on Zoom to talk about something important after the break?" What is it like to be on the other side of a Data Colada Zoom call, and how much does academic fraud contribute to the increasingly negative public opinion of universities in general? I am Steven Dubner. This is Freakonomics Radio.

We'll be right back. Okay. So Max Bazerman, a senior researcher and highly respected professor at Harvard Business School, receives an email asking him to meet with the Data Colada team, and they say it 's important. Then Zoom happens and the first part of the meeting is, uh, the Data Colada team showing me the evidence of fraud in the insurance article. And, um, it's kind of overwhelming. These guys are careful and meticulous, and they convinced me that there was fraud in the study. This study on insurance was one of three studies in the Science article that Bazerman had co-authored years earlier.

Researchers at Data Colada examined the data that Dan Arieli had used and found several suspicious things. The most obvious was a data graph called a histogram, showing the number of miles driven each year by the people in their study. For data like this, a histogram typically looks like a bell curve, with many people clustered around the center and some outliers veering toward the extremes. But this histogram showed a nearly uniform distribution of drivers from 0 to 50,000 miles. This is not what real data looks like , the Data Colada team wrote in their blog post.

And we can't think of a plausible and benign explanation for it. And what is Max Bazerman thinking now? I am simply overwhelmed by the fact that I am the author of a fraudulent article. Later, new information emerged from the insurance company that had provided the data to Dan Ariely. They told the Planet Money podcast that the data Ariely published was significantly different from what they had given her. And in their original data, there was no difference between those who signed the forms at the top and those who signed at the end.

Although Ariely declined to give an interview for this episode, he sent a written statement. As someone who has spent many years studying dishonesty, he wrote, I appreciate the irony of being accused of dishonesty. There is no doubt that the data that underpinned the 2012 study, which I co-authored with four other researchers on dishonesty, were , well, dishonest. I diligently tried to understand what went wrong, but since this happened more than 15 years ago, I can't say for sure what occurred. He added: " All five co-authors of the study in question participated in review sessions with people from the insurance company and asked questions about the data.

In the end, we were all satisfied with the answers we received and collectively decided to move forward with the article." When we shared Ariely's statement with Max Bazerman, he replied: " I am extremely confident that I never participated in such review sessions, and I have confirmed this in conversations with Lisa Shu and through a thorough search of my email records." The final part of Ariely's statement reads: " The circumstances that led to the falsification of the data are being investigated by Duke University. I am confident that the investigation will find no evidence to suggest that I was responsible for any data manipulation.

I am sure that this matter will be behind us very soon and that I will resume my research at Duke full steam." That concludes Ariely's statement. As for the journal PNAS, they finally retracted the original article . Meanwhile, on that Zoom call, the Data Colada team had more news for Max Bazerman. So, after they presented me with the insurance company's evidence, they said, "And now, the worst news," which is when they introduced the allegation of data fabrication in one of the lab studies. "One of the laboratory studies," meaning one of the separate studies in the same article whose data came from Francesca Gino; Data Colada said they also found serious problems with her data and, to make matters worse, there was evidence of data fabrication in three other projects that Gino co-authored.

And when you say they said " now, the worst news," I suppose, but correct me if I'm wrong, that they said that because they know you had a long and close relationship with Francesca Gino. Yes. And I was clearly more drawn to those laboratory studies. And you are a co-author on more than one of the articles. I was only a co-author of one of the four articles they were showing me, but at that time, I had already co-authored eight different empirical studies that had Francesca Gino as a co-author.

So, what happened next in the Zoom call? So they provided the evidence, and I became aware of what Data Colada later called a "cluster of forgeries," that there are at least two frauds in the same article, or that this was likely the case . And then Data Colada basically said, "So Max, you 're the Harvard professor; we think Harvard should have access to this information. You're the person who's going to bring it to them?" And I said, " No, thank you," because why? Because I certainly thought Harvard should be aware of what I was seeing, but I didn't want to play a central role in making that happen.

Data Colada began investigating Francesca Gino after receiving a tip from a graduate student named Zoe Zeani and another anonymous researcher. In addition to the article about signatures, the Data Colada team wrote, and we quote, " we believe that many other articles authored by Gino contain false data , perhaps dozens." Professor Gino indicated that she had done nothing wrong, and we said that the data in those four articles contain evidence that strongly suggests the existence of fraud. This is Leif Nelson again, one of the three researchers at Data Colada, along with Yuri Simonson and Joe Simmons.

Filling the gap between these two positions is another entity, Harvard University. And they said that, although we didn't have —Joe, Yuri, and I didn't have access to any documents from their internal investigation; We only know what they said publicly, which is that they placed her on administrative leave and recommended the retraction of those four articles, or the retraction of three plus an amendment to a previously retracted article. And how confident are the researchers at Data Colada in the accuracy of their analysis? Here is Simonson. So, I would say that, regarding the falsity of the findings, I have no reasonable doubt.

Not long after Harvard Business School placed Francesca Gino on leave, she filed a lawsuit. So , we were sued along with Harvard for 25 million dollars. We were sued for defamation. Once again, Francesca Gino refused our interview request on a website called Francesca v. Harvard. She wrote: "I absolutely did not commit academic fraud. Harvard unfairly ruined my career. The only way to right this wrong is for me to sue Harvard." Regarding Data Colada," Gino wrote, "the decision to sue Data Colada was more difficult." "I have long admired the work of Data Colada.

I particularly respected their commitment to sharing any negative findings with the author before making the accusations public. However, in my case, Data Colada changed its procedure. Franchesca Gino also alleged in her lawsuit that Harvard Business School discriminated against her based on gender. And later, her allies wondered why Gino's punishment was so swift and severe when, for example, Harvard President Claudine Gay wasn't immediately disciplined for outright plagiarism in her research. Uh, so, as you probably know, Gay was recently forced to resign as president. More academic fraud in the headlines.

I asked Leif Nelson how he felt when he learned that Franchesca Gino was suing him and the other members of Data Colada. Certainly frightening. Frightening because it's something very unfamiliar. I found out through conversation; basically, I was exchanging emails with a reporter, and between those emails, she came back to me and said, 'Well, now, given the lawsuit, would you like to add a...'" New comment? "And I basically said, 'What?'" Which process are you talking about? "And it's a devastating thing to think, 'Oh my God, it's like the whole house is falling apart and nobody warned me.' Look, I definitely had moments after that where I was scared for myself and my family.

And that's Joe Simmons. Like, just the amount of money involved. I didn't quite grasp it at first. I mean, that 's not how you judge these things. There are a million chances to prove her wrong. A million. Like before, many. There are many chances. And that's what you're going to do. You're going to sue three individuals for $25 million . That sounds like it doesn't sound good. The Data Colada team found that it 's expensive to defend yourself in a lawsuit like this. Some colleagues created a GoFundMe campaign.

Here's Simonson. In 24 hours, they had $200,000. We found a First Amendment lawyer who is representing us. We learned a lot of the annoying things that happen with lawyers that you don't see on TV shows, like deadlines , language, and the time the judge takes." Things take forever. Uh, I mean, it makes academia seem efficient in comparison. It's not only good to have money, but it's also good to know that thousands of people are willing to support, at least publicly, what you're doing. So, that was a big incentive.

My first thoughts were: my God, how is anyone going to do this again? If you can be prosecuted for conducting this kind of research, and we are professors at business schools in, you know, very good institutions that have a lot of resources, we are certainly in a position to withstand this better than, say, the average person in the field. And therefore, just the inhibiting effect on scientific research and criticism, it's as if everything we've worked for for 10 years is now lost? Our field doesn't have a culture of open criticism.

It's not considered acceptable, and that's Simine Vazire. If I cite a specific theory or discovery , that's considered a personal attack on the people associated with that theory or discovery, even if I do n't talk about the people behind it, and that's considered unacceptable. Vazire is a professor of psychology. at the University of Melbourne. She is also the new editor-in-chief of the important journal Psychological Science and has already been a central figure in the movement to reform the behavioral sciences. It was Vazire who created the GoFundMe campaign for the Data Colada team after Francesca Gino sued them.

I think one thing we can say for sure is that humans are very good at deceiving themselves. There are many reasons why researchers want to believe they have found the answers to these problems, right? One is a pro-social reason. They want to help solve these problems. They want to help people. Another is more selfish; they want a seat at the policymaking table. They want attention for themselves. They want to promote their theory and their brand. And it's also a matter of survival. To remain in academia, to be able to continue doing research, you need to have successes.

And those successes often mean selling your work, and sometimes selling your work beyond measure. We want to be taken seriously as scientists and to be scientific. And that means being calibrated and careful, without overdoing it. But at the same time, the people who Those who exaggerate will probably get more of those successes that bring them attention, give them a seat at the table, secure them the next grant, the next job, and so on. When you describe incentives in this way, it seems to me that these incentives conspire against the scientific method, doesn't it?

Yes. So, if you were just a rational agent acting in the most selfish way possible as a researcher in academia, I think you would cheat. I think that's exactly how incentives are structured. I don't think most people do that, but not because of incentives . So, the more I hear you talk about these, let's say, perverse or at least mixed incentives that conspire against the pure pursuit of knowledge even in academia, I think that if people like you, if fellow scientists, can't successfully evaluate these claims, what should the public do?

What should we think when we hear a claim that is often amplified by the media? Doesn't that make everyone skeptical or even cynical about everything? I think perhaps this is appropriate for areas like psychology, where we are dealing with very... Confusing and complicated things that are determined in various ways, that have many causes. It's difficult to measure even one of the causes. And therefore, I think if you hear a statement that seems too good to be true or too simplistic, I think it's appropriate to use common sense to be skeptical of it.

Generally, I think we're taught that science can overcome common sense and that you shouldn't simply discredit science just because your common sense goes against it . But I think that should be different for different sciences, and psychology is much more difficult to do and is a relatively young science. We haven't even perfected the measurement of many of these concepts yet, much less all the other steps to study them. Do you think your field is in crisis ? I think so. What kind of crisis? I mean, we hear the expression "replication crisis," which is a specific response to the fear that research is rubbish.

But I assume it's broader than that . Yes. I mean, I think, to our credit, it's broader than that. I think our field is really in a period of intense self-examination. This It will show the world, I believe, how committed we are to scientific values, the way we've handled this crisis. There are so many ramifications to this crisis. And I think it 's just a crisis of, I don't know, integrity or credibility, or whatever the most fundamental word you can use to describe what it means to be scientific is.

There was a lot of debate at the beginning of the crisis about whether we should air our dirty laundry in public. The people who argued against that, I lost a lot of respect for them. But the other side won. We aired our dirty laundry in public. And I think we deserve some credit for that. I don't think we should rest on our laurels. Identifying the problem isn't the same as changing our practices. We should do a self-analysis of how we got there and how we can prevent it from happening again.

Max Bazerman took the lead in this self-analysis. You know, I've been involved in so many research projects where, as the most senior member of the team, I never looked at the database. So, you know, does that make me guilty of something? I think so. I think it makes me complicit for not having done so. more verifications, because I trusted it completely. And I think I should, in a way, be held responsible for not having done a better job of verification, not just on the article about signatures, but on my research more broadly.

And, you know, on the article about signatures, I was bothered by some aspects of the data and asked a lot of questions back in 2011, but I ended up getting an answer that I wanted to be true, I accepted it, and I didn't look at that database back then either. Would I have liked to look? Absolutely. I 'm not sure if this story would be unfolding today if I had looked at that data in 2011. There are so many positive influences that the social sciences can have, from leading us to eat healthier foods to exercising more and saving for our retirement.

And social scientists have been extraordinarily good at helping to figure out how to steer people in the right direction. And the vast majority of that work is honest and reliable research that we should pay attention to. And if we now end up with fraud in the social sciences as big news , then all the reliable material will have less value and less impact than it should. Next time on this show, we'll continue this conversation, but from different angles, including the money. You've probably heard of diploma mills or puppy mills, but what about scientific paper mills?

They can range from hundreds of dollars to thousands of dollars per paper. And they're publishing tens of thousands, and sometimes even more, papers per year. So you can start doing the math. We'll start, maybe even finish. That will be next time on this show. Until then, take care of yourself and, if you can, take care of someone else too . By the way, as we were finishing this episode, someone very close to me turned 64. And in response to the questions posed by Paul McCartney in that song, the answer is yes.

Freakonomics Radio is produced by Stitcher and Renbud Radio. All of our archive since 2010 can be found on any podcast app or at freakonomics.com, where we also publish transcripts and notes from the episodes. This episode was produced by Alina Coleman. Our team also includes Elellanor Osborne. Elsa Hernandez, Gabriel Roth, Greg Ripen, Jasmine Clinger, Jeremy Johnston , Julie Caner, Lyric Bowitch, Morgan Levy, Neil Kuth, Rebecca Lee Douglas, Ryan Kelly, Sarah Lily, and Zack Leinsky. Our theme song is "Mr. Fortune" by The Hitchhikers. All other music is composed by Luis Gera.

As always, thank you for listening. So, if I were to rephrase your answer, you basically said that podcasts are better than academic journals . Correct. As a former podcast host, I have to agree. Freakonomics Radio Network, The Hidden Side of Everything. Stitcher.