Experiments and Complexity
December 17, 2015
What is desirable for science and what is desirable for the humans who carry out scientific research can diverge. Humans, according to Adam Smith, have an instinct for perfect order, which we conflate with beauty.
Hence, we buy fancy and expensive watches, even though we don’t get any extra utility from information on time. What counts is the attainment of the perfection of a device for keeping time.
The same thing may be happening in the area of field experiments in economic development. Scientists love the supposed beauty of intricacy and perfection. The result, according to Chris Blattman, is that experiments are becoming increasingly (and unnecessarily) more complex:
“Every year the technical bar gets raised. Some days my field feels like an arms race to make each experiment more thorough and technically impressive, with more and more attention to formal theories, structural models, pre-analysis plans, and (most recently) multiple hypothesis testing. The list goes on. In part we push because want to do better work. Plus, how else to get published in the best places and earn the respect of your peers?
“It seems to me that all of this is pushing social scientists to produce better quality experiments and more accurate answers. But it’s also raising the size and cost and time of any one experiment.
“This should lead to fewer, better experiments. Good, right? I’m not sure. Fewer studies is a problem if you think that the generalizabilty of any one experiment is very small. What you want is many experiments in many places and people, which help triangulate an answer.”
I agree. Taking things to their logic extremes satisfies some part of the intellect and emotion for perfection, but closes off lots of avenues of exploration. Let a thousand flowers bloom!*
[* Misquotation of Mao, who encouraged open dialogue by saying “Let a hundred flowers blossom.” When dissidents came forward, many were executed. I draw on Mao’s quote, but not the stifling of opinion.]