Moving past the debate about randomized experiments May 25, 2010
Posted by Paul Duignan in : Doing evaluation more efficiently, Easy Outcomes, Evaluation debates, Evaluation planning, Impact evaluation, Using the approach , trackbackA colleague Bob Williams recently drew attention to articles on the New Yorker about the use of randomized experiments and particularly one from an economist advocating their widespread use in a range of program areas.
I’ve been involved in a number of seemingly endless discussions and presentations about the pros and cons of randomized experiments and the rise of what are being called the Randomistas – those advocating for a much wider use of randomized experiments. In this post I want to talk about how we can move beyond these seemingly endless discussions.
In my experience, the outcome of such discussions inevitably are the following points made by the non-Randomistas: 1) randomized controlled trials have their place in certain situations; 2) they are not suited for determining causality in many other cases; 3) it is irritating that at the current time there are a number of people out there who are pushing randomized controlled trials as the ‘gold standard’ for impact evaluation with either an ignorance of their inevitable limitations or a willful unwillingness to be realistic about their limitations; 4) in the hands of people who do not understand anything about the technical issues involved the simplistic demand for more randomized controlled experiments is likely to lead to misallocation of evaluation resources to overambitious randomized experiments which may fail in the course of the experiment (and hence waste resources); lead to misleading results; or eventuate in a systematic bias in program delivery toward those programs which are easier to subject to randomized experiments.
Randomistas on the other hand will argue: 1) they acknowledge that there are circumstances where randomized experiments will not be appropriate; 2) however, randomized experiments are incredibly powerful (when they can be done properly) at determining causality and their being able to establish this is incredibly important in a number of cases to cut through the current confusion and political agendas which surround decision-making about resource allocation; 3) we should be pushing everyone to think very hard about when randomized experiments can be done so that we do not miss any opportunities to do so.
At the end of such discussions I generally find myself unfortunately in general agreement with all of the above points!
In the light of ending up at the end of discussions like this feeling that way, the recent focus of my work has been on providing a practical way of moving forward in the face of the inevitability of discussions like the above. Ones from which no one ‘side’ will emerge the clear ”winner’. The end result of which is a lot of people spending a lot of time talking past each other.
The methodology I think we should use to move beyond endless discussions like this can be termed Impact Evaluation Design Assessment and it is a component of my Easy Outcomes approach.
This approach starts from first acknowledging the Randomistas point that it is worth at least a little thought in regard to any program to consider whether a randomized controlled experiment could be done on it. However such an assessment should be put in the context of also considering the other alternative impact evaluation designs to randomized experiments.
This leads to a relatively straightforward recommendation that any program assess a list of possible impact evaluation designs. I have a list of seven of them that I use which, importantly, includes a full range of impact assessment methods. The appropriateness, feasibility and affordability of each one of these design types should be documented as part of any monitoring or evaluation planning for any program. It seems to be that this gives us a practical tool which does justice to both sides of the debate while moving us forward beyond just a permanent state of discussing this issue.
The article which shows how to do it is Duignan, P. (2009). Impact evaluation – when it should, and should not, be used. Outcomes Theory Knowledge Base Article No. 242. ( http://knol.google.com/k/paul-duignan-phd/impact-evaluation-when-it-should-and/2m7zd68aaz774/86 ). The list of seven impact evaluation design types is Duignan, P. (2005-2009). Impact/outcome evaluation design types. Outcomes Theory Knowledge Base article No. 209. ( http://knol.google.com/k/paul-duignan-phd/seven-possible-outcomeimpact-evaluation/2m7zd68aaz774/10 ).
An example of one page of the analysis within a visual evaluation plan is here http://www.outcomesmodels.org/models/communitycentral31-slices/communitycentral31-10.7.html .
Instructions on how to build visual evaluation plans like this are at http://knol.google.com/k/paul-duignan/-/2m7zd68aaz774/134
To do this type of analysis done for all programs would require some capability development around knowledge of impact evaluation but I don’t see that it is unrealistic to try to move to a point where it is done in regard to every program in some form. Or it might be done more generally for certain types of similar programs and individual programs could just point to the analysis.
Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).
Comments»
Hi Paul,
I enjoyed your blog post.
It’s amazing to me how the word “rigor” is still equated with randomized control trials and quasi-experimental designs, even after Lincoln and Guba so diligently described how to ensure reliability and validity regardless of the evaluation approach or method. I see it most decisively as I assist organizations with writing the evaluation plans for grant applications. It most certainly still rings true with who you call the Randomistas.
I believe that using outcomes planning approaches as you describe would help clear the air and assist both the Randomistas and Non-Randomistas in coming to terms with doing what is valuable for the project instead of having tunnel vision.
Hi Michelle,
We still need to catch up for that talk some time! Sorry I have been busy with many things including writing a first draft of a book about my work.
As you say, I think that we need some way of working across the different perspectives, otherwise people are just going to keep talking past each other.
There is powerful growth of the Randomista approach which is coming in at least part from people who would not call themselves evaluators, economists in particular.
I think that as the evaluation profession went through a long period of thinking about different impact evaluation designs and methodologies, that that process needs to happen now on a much larger scale.
In particular more sophisticated thinking, for instance, that in some situations you are better off doing smart formative evaluation (for those not familiar with the term – evaluation to ensure that a program is well formed – sometimes also called implementation evaluation) than attempting to do an impact evaluation where it is not appropriate, feasible or affordable.
It is great that so many people are getting into evaluation, evaluators just need to look to engage all of the different people now thinking intensely about evaluation.