Psychology · Level 5 · 208 words

When Results Do Not Repeat

Original passage © Studio AM, written for Fluency.

Around 2011, psychology began to audit itself, and the audit did not go well. A large collaboration repeated one hundred published studies, using larger samples and the original methods where those could be obtained. Fewer than half produced a result that reached statistical significance the second time, and the effects that did survive were, on average, roughly half the size first reported.

The temptation is to read this as a story about fraud. It is mostly a story about incentives. Journals preferred surprising, tidy findings; careers were built on them; and a researcher holding many measures and a flexible analysis could, in complete sincerity, wander toward the version of the data that worked. Nobody has to lie for a research literature to fill with results that are true only of the sample that produced them.

The response has been unglamorous and largely structural: preregistering hypotheses before the data are collected, publishing methods that are accepted before any results exist, sharing raw data, and running samples large enough to detect the modest effects that human behavior actually offers. The irony is worth holding onto. A field that lost public trust by overstating what it knew has recovered some of it by publishing, in detail, the record of being wrong.

Comprehension questions

Choose an answer, then check your work. Nothing is saved or sent.

4 questions
1. Which statement best captures the argument of the passage?

Show answer for question 1

C. Many published findings failed to repeat, largely because of incentives and flexible analysis, and the field answered with structural reform.
The passage rejects fraud as the main cause, names incentives instead, and devotes the last paragraph to structural fixes.

2. What happened to the effects that did survive the repeat attempts?

Show answer for question 2

A. They were on average about half the size first reported.
The first paragraph gives this figure directly, alongside the count that reached significance.

3. What is the irony the passage points to at the end?

Show answer for question 3

B. That trust lost by overstating knowledge was partly regained by publishing a detailed record of error.
The closing sentence sets losing trust through overstatement against recovering it through openly documented mistakes.

4. In this passage, sincerity means:

Show answer for question 4

C. honest belief, with no intention to mislead
The phrase in complete sincerity is what separates this behavior from the fraud the passage has just ruled out.

Source: Written for Fluency. Original passage © Studio AM, written for Fluency.