Ease of impact evaluation and bias within ‘what works’ evidence-based practice systems January 4, 2009
Posted by Paul Duignan in : Attribution, Evaluation planning, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , trackback
Back on deck after Christmas and turning the mind back from eating and having a great time with relatives to outcomes and evaluation again. I’ve just put up an article in the Outcomes Theory Knowledge Base about the issue of evidence-based practice systems not taking into account variations in the ease of undertaking impact evaluation between program types.
There are many examples of evidence-based practice systems being set up (two classic examples are the Cochrane Collaboration in health care and the Campbell Collaboration in the social policy area) which are attempting to use information from impact evaluation (evaluation which demonstrates attribution of improvements in high level outcomes to particular programs). I think that such initiatives are generally a great idea. My article deals with a particular situation which sometimes arises which needs to be addressed in any well constructed ‘what works’ system.
This issue is that in a ‘what works’ system in an ideal world all of the programs which are being compared will have a similar ease of undertaking impact evaluation. This is roughly the situation in an area like therapeutic drugs where evidence-based systems relying on aggregating the results of impact evaluation studies (using meta-analysis) have been so successful in improving health outcomes.
However, in many other sectors, there may be a wide variation in the ease of undertaking impact evaluation between program types – for instance, individual interventions in controlled settings may be much easier to evaluate for impact than community programs in fluid community settings.
‘What works’ systems need to take this into account if they are not to be biased in the direction of programs for which impact evaluation is relatively easy to undertake. As we move forward with the development of evidence-based system across many sectors, this is the type of nuanced approach we are going to have to take if we are going to build sophisticated systems which give us accurate information for evidence-based practice. How we can develop such systems is set out in the article. Check it out.
Paul Duignan PhD
Comments»
no comments yet - be the first?