Randomistas Rule August 16, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Evaluation debates, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , trackbackJust read and commented on an interesting article referred to on the 3IE site – a site dedicated to improving evidence about what works in international development. The article was by Martin Ravallion and was about the rise of the Randomistas in international development economics. Randomistas are those who promote much more use of randomized trials to try and work out what works in international development. It is a good article which points out the fact that randomized trials are not feasible in many important types of development interventions. This debate is the same one which is occurring in many sectors at the moment and one which has been debated on and off in the evaluation field for many years. My take on it is that we need to develop some underlying principle which we can debate and generally agree on so that we don’t need to have this debate endlessly without seemingly making much progress on it.
My concern is that these debates are conducted in a rather ad hoc manner, sector by sector, and sometimes within disciplinary silos. In the development of outcomes theory what I’m trying to do is to specify a set of generic principles which can be debated and then, hopefully, generally accepted to provide a sound theoretical basis for this sort of discussion (so we don’t need to have the same discussion time and time again – for anyone familiar with the literature of the evaluation discipline it is a familiar debate).
The outcomes theory principle being discussed here is one called the Equivalent Ease of Impact Evaluation Principle. This principle states that a rational decision-maker will only use the results from impact evaluation (or as in this case, just one type of impact evaluation – randomized trials) as the main determining factor when deciding between alternative interventions when these interventions are very similar in terms of the easy with which impact evaluation can be carried out. To use impact evaluation results as the major determinant of how you act in a situation where the Equivalent Easy of Impact Evaluation Principle does not hold is to act as an irrational decision-maker. This is set out in more detail in Duignan, P. (2009). Implications of an exclusive focus on impact evaluation in ‘what wo….
So, in the area of pharmaceuticals, the Equivalent Ease of Impact Evaluation Principle generally applies in regard to a large number of drug types – it is therefore rational to use the results of impact evaluation (randomized controlled trials) to determine which drugs to prescribe. It is not rational to use the results of impact evaluations as the major determinant in deciding between, say, individual pharmaceutical interventions compared to community-wide non-drug interventions (on which it will usually be harder to undertake randomized controlled trials).
So in the international development area, due to the issues raised in the article, the Equivalent Ease of Impact Evaluation Principle does not apply and therefore just the results of impact evaluations cannot be used as the major determinant when deciding between which interventions to fund. In the world in which I would like to operate. We would ask people whether or not they agree with whether this principle does or does not hold in regard to international development and the debate would be conducted at that level.
Of course, accepting that the Equivalent Ease of Impact Evaluation Principle does not hold in the international development area does not mean that it is irrational to argue: 1) that not enough randomized trials have been done up until now; 2) that randomized trials should be actively promoted at the current time because there has been an evaluation culture in international development which has not paid enough attention to the possibilities of doing them; 3) that we should not try to push the boundaries as to current thinking as to what can, and cannot, be subject to a randomized trial; or; 4) that the results of randomized trials do often provide useful information.
Paul Duignan
OutcomesTheory.org
OutcomesBlog.org
Folllow me on Twitter Subscribe to my E-newsletter
Comments»
no comments yet - be the first?