What can be claimed about whether a program works or not from a logic model? January 13, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Impact evaluation, Outcomes models, Uncategorized , trackback
We sometimes hear things like: “a logic model was used to show that the program works”. I’m interested in tidying up such talk so that we are very clear about exactly what it is that is being claimed in regard to showing whether or not a program works. I’ve put up a new article in Knol which deals with the question of the types of claims we can make in regard to logic models (I call them outcomes models) and the types of arguments we can mount about whether or not programs work based on these claims. The article is here.The article is rather intense, but I think that it’s important that, as evaluators, we get on top of this sort of thing. In the article, I set out three claims that can be made in regard to logic models (or sub-parts of logic models). These are:
- Claim 1. that they represent how it is thought that a program will work (without providing ‘evidence’ in regard to the links)
- Claim 2. that they represent how it is thought that a program will work and that this is supported by evidence from other programs. (Note: evidence is not defined here, I don’t want to get into that debate in this analysis – for our purposes here, it is whatever stakeholders will accept as evidence).
- Claim 3. that they represent what has been demonstrated to be occurring in the case of a particular program.
Logic models based on these three claims can then be used in various ways to attempt to mount an argument that stakeholders should accept that a program is causing high-level outcomes to occur. These arguments run along a continuum of ‘confidence’ in the belief that the program is causing high-level outcomes to occur. In summary, they are:
Level of Confidence 1
Argument A. That because what we believe is happening has been laid out transparently, stakeholders should have more confidence that the program is causing high-level outcomes to improve (based on a model making claim 1).
Level of Confidence 2
Argument 2. That because we have laid it out and it is evidence-based stakeholders should have even more confidence that the program is causing high-level outcomes to improve (based on a model making claim 2).
Argument 3. That because we have demonstrated that lower-level parts of the model are occurring in the case of a particular program (based on a model making claim 3 in regard to its lower levels) and we have laid out transparently what we believe is happening (based on a model making claim 1) stakeholders should have confidence that the program is causing high-level outcomes to occur.
Level of Confidence 3
Argument 4. That because we have demonstrated that lower-level parts of the model are occurring in the case of a particular program (based on a model making claim 3 in regard to its lower levels) and we have an evidence-based model connecting the lower levels to the higher-levels (based on claim 2), stakeholders should have confidence that the program is causing high-level outcomes to occur.
This analysis does not imply that stakeholders will, or ought to, accept that the level 3 argument (or any other level argument) constitutes attribution having been established in the same way as it is done in some outcome/impact evaluation designs. The purpose of this analysis, and much of the focus of my conceptual work with outcomes theory, is to help us be more explicit about exactly what claims and arguments people are mounting so that stakeholders can make their own judgements about those claims. As I said at the top of this post, at the moment I think that it is all rather loose. E.g. “we used the logic model to justify the program” or “a program theory approach was used to show that the program worked”.
Paul Duignan, PhD
Comments»
The warm feeling I get when someone is thoughtful enough to say thank you for having been helped far outweighs the empty one I get when there’s no feedback at all.
______________________
Drug Intervention