jump to navigation

Moving past the debate about randomized experiments May 25, 2010

Posted by Paul Duignan in : Doing evaluation more efficiently, Easy Outcomes, Evaluation debates, Evaluation planning, Impact evaluation, Using the approach , trackback

A colleague Bob Williams recently drew attention to articles on the New Yorker about the use of randomized experiments and particularly one from an economist advocating their widespread use in a range of program areas.

I’ve been involved in a number of seemingly endless discussions and presentations about the pros and cons of randomized experiments and the rise of what are being called the Randomistas – those advocating for a much wider use of randomized experiments. In this post I want to talk about how we can move beyond these seemingly endless discussions.

In my experience, the outcome of such discussions inevitably are the following points made by the non-Randomistas: 1) randomized controlled trials have their place in certain situations; 2) they are not suited for determining causality in many other cases; 3) it is irritating that at the current time there are a number of people out there who are pushing randomized controlled trials as the ‘gold standard’ for impact evaluation with either an ignorance of their inevitable limitations or a willful unwillingness to be realistic about their limitations; 4) in the hands of people who do not understand anything about the technical issues involved the simplistic demand for more randomized controlled experiments is likely to lead to misallocation of evaluation resources to overambitious randomized experiments which may fail in the course of the experiment (and hence waste resources); lead to misleading results; or eventuate in a systematic bias in program delivery toward those programs which are easier to subject to randomized experiments.

Randomistas on the other hand will argue: 1) they acknowledge that there are circumstances where randomized experiments will not be appropriate; 2) however, randomized experiments are incredibly powerful (when they can be done properly) at determining causality and their being able to establish this is incredibly important in a number of cases to cut through the current confusion and political agendas which surround decision-making about resource allocation; 3) we should be pushing everyone to think very hard about when randomized experiments can be done so that we do not miss any opportunities to do so.

At the end of such discussions I generally find myself unfortunately in  general agreement with all of the above points!

In the light of ending up at the end of discussions like this feeling that way, the recent focus of my work has been on providing a practical way of moving forward in the face of the inevitability of discussions like the above. Ones from which no one ‘side’ will emerge the clear ”winner’. The end result of which is a lot of people spending a lot of time talking past each other.

The methodology I think we should use to move beyond endless discussions like this can be termed Impact Evaluation Design Assessment and it is a component of my Easy Outcomes approach.

This approach starts from first acknowledging the Randomistas point that it is worth at least a little thought in regard to any program to consider whether a randomized controlled experiment could be done on it. However such an assessment should be put in the context of also considering the other alternative impact evaluation designs to randomized experiments.

This leads to a relatively straightforward recommendation that any program assess a list of possible impact evaluation designs. I have a list of seven of them that I use which, importantly, includes a full range of impact assessment methods. The appropriateness, feasibility and affordability of each one of these design types should be documented as part of any monitoring or evaluation planning for any program. It seems to be that this gives us a practical tool which does justice to both sides of the debate while moving us forward beyond just a permanent state of discussing this issue.

The article which shows how to do it is Duignan, P. (2009).  Impact evaluation – when it should, and should not, be used. Outcomes Theory Knowledge Base Article No. 242. ( http://knol.google.com/k/paul-duignan-phd/impact-evaluation-when-it-should-and/2m7zd68aaz774/86 ). The list of seven impact evaluation design types is Duignan, P. (2005-2009). Impact/outcome evaluation design types. Outcomes Theory Knowledge Base article No. 209. ( http://knol.google.com/k/paul-duignan-phd/seven-possible-outcomeimpact-evaluation/2m7zd68aaz774/10 ).

An example of one page of the analysis within a visual evaluation plan is here http://www.outcomesmodels.org/models/communitycentral31-slices/communitycentral31-10.7.html .

Instructions on how to build visual evaluation plans like this are at http://knol.google.com/k/paul-duignan/-/2m7zd68aaz774/134

To do this type of analysis done for all programs would require some capability development around knowledge of impact evaluation but I don’t see that it is unrealistic to try to move to a point where it is done in regard to every program in some form. Or it might be done more generally for certain types of similar programs and individual programs could just point to the analysis.

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

Comments»

1. Michelle Baron - May 27, 2010

Hi Paul,

I enjoyed your blog post.

It’s amazing to me how the word “rigor” is still equated with randomized control trials and quasi-experimental designs, even after Lincoln and Guba so diligently described how to ensure reliability and validity regardless of the evaluation approach or method. I see it most decisively as I assist organizations with writing the evaluation plans for grant applications. It most certainly still rings true with who you call the Randomistas.

I believe that using outcomes planning approaches as you describe would help clear the air and assist both the Randomistas and Non-Randomistas in coming to terms with doing what is valuable for the project instead of having tunnel vision.

2. Paul Duignan - May 27, 2010

Hi Michelle,

We still need to catch up for that talk some time! Sorry I have been busy with many things including writing a first draft of a book about my work.

As you say, I think that we need some way of working across the different perspectives, otherwise people are just going to keep talking past each other.

There is powerful growth of the Randomista approach which is coming in at least part from people who would not call themselves evaluators, economists in particular.

I think that as the evaluation profession went through a long period of thinking about different impact evaluation designs and methodologies, that that process needs to happen now on a much larger scale.

In particular more sophisticated thinking, for instance, that in some situations you are better off doing smart formative evaluation (for those not familiar with the term – evaluation to ensure that a program is well formed – sometimes also called implementation evaluation) than attempting to do an impact evaluation where it is not appropriate, feasible or affordable.

It is great that so many people are getting into evaluation, evaluators just need to look to engage all of the different people now thinking intensely about evaluation.


canadian cialis online
buy doxycycline capsules
viagra 6 free samples
viagra for sale in the uk
cipro pregnancy
cipro with food or without
vet med doxycycline
doxycycline 75
doxycycline info
buy doxycycline no prescription from uk
lumigan online no perscription overnight
www.viagra.com
where can i get cialis
get viagra
lumigan and alphagan
buying cialis online
discounted cialis online
prescription antibiotic relative strength doxycycline
order cialis canada
generic name for lumigan
cialis without prescription
how to buy viagra in%
viagra alternatives
lumigan headaches
ciprofloxacin dosage gastroenteritis
reactions to the drug doxycycline hyclate
fda approved cialis
doxycycline in las vegas nevada usa
viagra canadacialis overnight shipping
buy doxycycline next day delivery
pfizer viagra no prescription
pediatric zithromax
free trial of cialis
side effects doxycycline puppies
doxycycline 1 month
cialis canadian cost
viagra fast delivery
buy viagra now online
lumigan ndc
usa viagra sales
cipro hc otic contraindications
we deliver to canada viagra
lumigan yelp
best prices viagra
doxycycline nausea
buy viagra 100mg
cipro prednisone drug interactions
lumigan bottle size
gums doxycycline clindamycin
lumigan rc side effects
ciprofloxacin joint pain
low cost viagra uk
doxycycline kidney disease
dizziness from doxycycline
ciprodex for ears
lumigan drops
doxycycline mechanism of action
cialis quick
purchase real name brand viagra
lyme disease doxycycline dose pediatric
purchase zithromax
viagra uk fast delvery
viagra costs
cialis iop
viagra rx in canada
where can i get viagra online from canada
viagra tablets sale
doxycycline 20 mg tablets
rx generic viagra
cialis 20mg pricing
buy viagra in australia
doxycycline hyclate capsules
viagra 100 mg
doxycycline 500
buy viagra overnight delivery
best prices on viagra
should i chew cialis
doxycycline xopenex
buy lumigan on line without a perscription
osteomyelitis and doxycycline
viagra alternative
lumigan rxlist
lumigan cost
cialis online ordering
can i buy viagra in canada
where to buy lumigan no perscription no fees
best prices on brand viagra
viagra, candadian drugs
how does doxycycline work on cats
doxycycline wiki
viagra us pharmacy
zithromax sinusitis
lumigan vs lumigan rc
cialis without prescription
viagra for her
viagra canada fast shipping
buy original pfizer viagra
and had that things anywhere drug cheap viagra 100mg
generic viagra canada
cialis professional 100 mg
cialis prices
cialis woman
lumigan dosage form
cipro prostatitis time
cheap viagra pills online
doxycycline and bruising
cipro hc generic form
order cialis on internet
buy cialis on line no prescription
cialis side effects
us discount viagra overnight delivery
australia viagra onlineorder viagra online no prescription
cialis 10mg
purchase cialis overnight delivery
bruising on cialis
cialis professional no prescription
lumigan generic buy
doxycycline sunburn reaction treatment
purchase viagra canada
cialis 5mg brand name
cialis price in canada
health center for viagra prescriptions
lumigan generic canada
viagra cialis levitra canada
buy lumigan online without a perscription
buy cialis online without prescription
doxycycline and breathing
how to buy viagra in canada
viagra best online store canada
buy viagra 10 pills
lumigan generic india
doxycycline side effect stress worry
buy doxycycline chlamydia
buy ciprodex otic
canadian pharmacy online cialis
cipro side effects neuropathy
buy real cialis
lumigan in pregnancy
doxycycline canine
buy doxycycline without
cheapest viagra online
canadian healthcare viagra sales
doxycycline on empty stomach
lumigan hypertension
cialis for daily use
brand cialis
find cialis online
cialis generic recommended
cialis no prescription needed
levitra versus viagra
real viagra canada
buy cialis online china
find cheap viagra online
lumigan price comparison
buy doxycycline in canada
lumigan kidney disease
discouont viagra
t pallidum treponema doxycycline tetracycline
viagra no prescription
zithromax uti
cipro joint pain side effects
cipro usage
best quality viagra
viagra tablets
lumigan rc canada
buy genuine cialis online
viagra pfizer no prescription
order viagra from vipps
cialis canada online pharmacy no prescription
best reviewed cialis sites
samples of cialis
doxycycline wiki
canadian pharmacy cialis pfizer
cialis online canada
buy viagra pills online canada
cialis 10mg price
cheap viagra online without prescription
zithromax pregnancy
own brand cialis
buy zithromax
doxycycline causing ceasure
cheap viagra online without prescription
cialis in usa
lumigan price singapore
cialis uk order
doxycycline youtube
cipro treatment infection
canadian cialisis
doxycycline yeast infection male
lumigan long lashes
doxycycline numbness
doxycycline hyclate
doxycycline monohydrate
non prescription viagra
doxycycline metabolism
genuine viagra online
viagra free pills
viagra online uk
online viagra canada
doxycycline birth control
viagra from canadian prescription
cialis super viagra
buy accutane online without dr approval
viagra online us pharmacy
canine doxycycline dose
cipro positive reviews
buy viagra from canadian pharmacy
lumigan with no perscription and delivered over night
drug stores canada viagra
buy ciprofloxacin ear drops
40mg cialis
cialis price
viagra sale
viagra shop
ciprofloxacin hcl warnings
cheapest price for viagra
buy doxycycline vietnam
canada prescription viagra
viagra canada 50mg
viagra online delivered next day
online generic cialis 100 mg
doxycycline urinary tract infection dosage
pharmacy selling viagra in israel