RCTs: the next big thing in politics?

The British physician Ben Goldacre, whose 2008 book Bad Science pushed him into the global front rank of science popularizers, has collaborated with a UK government thingie called the “Behavioural Insights Team” on an important paper whose title alone captures a significant amount of its importance. It’s called “Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials”. Goldacre refers to the paper as an attempt at writing a “Ladybird Book” on public-policy RCTs. I am not quite sure how to translate this for the New World audience: “Public Policy RCTs for Dummies” would be in the ballpark, though possibly in the left-field bleachers.

The randomized controlled trial is a model for knowledge-acquisition, a particular kind of scientific activity, that has progressed slowly upward in generality and power throughout modern times. It started in late 19th-century psychology and agriculture, took over medicine after the Second World War, and has recently (as Goldacre et al. point out) taken root in web design. Its history is a bit like the histories of Darwinian evolution or Turing machines or the equal-temperament keyboard: it’s an idea, created to address very particular situations or questions, that seems more and more fundamental the more people play with it.

Nobody would now think of introducing an important health intervention, whether it be a drug, a surgical procedure, or a piece of equipment, without subjecting it to an RCT. Heck, the Pepsi Challenge is a crude form of randomized controlled trial. The basic idea isn’t that complicated. So the question the Behavioural Insights people have been asking is, how come we almost never do this with public-policy interventions? Bureaucrats are deeply devoted to testing new ideas and programs by means of the “pilot project”, but rarely do they make sure such projects have the key elements of an RCT: (1) truly random assignment of the intervention being considered, (2) test criteria selected and announced in advance; (3) the existence of a matching control group that receives a different intervention or none at all; and (4) safeguards designed to limit the possibility of experimenter manipulation of results, whether conscious or unconscious.

One can imagine Sir Humphrey Appleby flinching at all this brute objectivity designed to deliver cold yes/no decision guidelines to the ruling class. Items 1, 2, and 4 supra deprive him of a lot of the power to get the result he wants in advance. And that is probably one reason why the RCT model has not been terribly influential in policy circles. Yet RCTs, as the UK paper notes, were embraced quickly in medicine despite being, in that setting, very expensive and sometimes fraught with ethical intricacies. When it comes to a question such as whether it is worthwhile for courts to follow up on parking tickets with text messages, RCTs can be absurdly cheap, and nobody could possibly find some other good argument for not doing one.

It’s easy to overstate the novelty of the public-policy RCT: the famous and still much-discussed RAND Health Insurance Experiment, for example, began more than 40 years ago. And entire fields of public policy are impossible to capture in an RCT lens. It would be nice if we could settle the macroeconomists’ intergenerational wrangling over the appropriate target for monetary policy by means of a controlled experiment, for example; but it is hard to see how you could pull that off. There are, however, fields of statecraft that are very open to RCTs, and that are in desperate need of the epistemic discipline they impose. The best example might be foreign aid, and the RCT model has caught fire amazingly quickly in that realm, thanks to scholars like Esther Duflo and Abhijit Banerjee.

Pepsi Challenge-ing economic-development interventions in the Third World has already won Duflo a Macarthur genius grant and the John Bates Clark Medal. There’s pretty much no top honour left for her to nab except the quasi-Nobel they give to economists, and she is a lock for that if she lives long enough. It also seems inevitable that some careers will be made preaching the RCT gospel here, in Canada; all that is needed is for a few good policy professionals to take up the torch. And look: Goldacre’s just handed it to you! He is, I think, a tremendously key figure in this development. For the past decade or so he has been doing for RCT methodology what pulp science fiction writers of the ’40s and ’50s did for physics, taking up the cause of Bradford Hill and Archie Cochrane and our own David Sackett to make lay converts for gold-standard evidence-gathering.

Maybe it’s even possible RCTs will turn up as an element in electoral politics, as a way of finessing ideological questions for post-Cold War voters who (I notice) don’t like labels and won’t take them. There have been a lot of jokes floating around about a post-ideological “Economists Party” devoted to accepted principles of free trade, public choice, and mainstream welfare economics. Perhaps it should be an “Evidence Party” instead?

Looking for more?

Get the Best of Maclean's sent straight to your inbox. Sign up for news, commentary and analysis.