Psychology · Level 5 · 208 words
When Results Do Not Repeat
Original passage © Studio AM, written for Fluency.
Around 2011, psychology began to audit itself, and the audit did not go well. A large collaboration repeated one hundred published studies, using larger samples and the original methods where those could be obtained. Fewer than half produced a result that reached statistical significance the second time, and the effects that did survive were, on average, roughly half the size first reported.
The temptation is to read this as a story about fraud. It is mostly a story about incentives. Journals preferred surprising, tidy findings; careers were built on them; and a researcher holding many measures and a flexible analysis could, in complete sincerity, wander toward the version of the data that worked. Nobody has to lie for a research literature to fill with results that are true only of the sample that produced them.
The response has been unglamorous and largely structural: preregistering hypotheses before the data are collected, publishing methods that are accepted before any results exist, sharing raw data, and running samples large enough to detect the modest effects that human behavior actually offers. The irony is worth holding onto. A field that lost public trust by overstating what it knew has recovered some of it by publishing, in detail, the record of being wrong.
Source: Written for Fluency. Original passage © Studio AM, written for Fluency.