Do RCTs suffer from a placebo effect?
The paper illustrates the point by undertaking two different RCTs on cowpea seeds in Tanzania. One is a traditional study where the control group knows they are getting traditional seeds and the treatment group knows they are getting modern seeds. The second is double blind; neither group knows what seed it is getting.
The traditional RCT shows a significant over 20% increase in yields from the modern seed. But the double blind RCT shows that virtually all of that improvement comes from changed behavior, not from any inherent effectiveness of the modern seed.
Specifically, the average treatment effect in the double blind RCT was zero! And when the harvests in the control groups across the two RCTs were compared, the blind control group showed a significant over 20% increase over the traditional RCT control group which knew they were getting the traditional seeds. This is the “pseudo-placebo” effect and it explains the entire average treatment effect in the traditional RCT.
In other words, the significant effect found in the traditional RCT was not due to better seeds, it was due to actions taken by the farmers who thought they were getting better seeds (they planted them in larger plots with more space between the plants on better quality land). These farmers’ expectations were wrong (in post experiment surveys, over 60% of them said they were disappointed in the yields), and the significant effect in the traditional RCT would not survive over time because the farmers, having adjusted their expectations downward would stop taking the actions that produced the “success”.
My takeaway is this: don’t use RCTs to identify good programs. Use them to weed out bad ones.