Distinguishing Evaluation from Monitoring (and other processes such as Performance Management and Assessment) January 30, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Systematic Outcomes Analysis, Using the approach , 5commentsI have blogged before about what I see as the wrong way to approach the problem of differentiating evaluation from related activities (e.g. monitoring, performance management, assessment). Last time it was about differentiating evaluation from quality assurance. The wrong way is to spend a lot of time on attempting to distinguish between processes by using high-level terms such as evaluation, performance management etc. A lot of ink and talk has been spilled on trying to do that over the years. In the case of monitoring and evaluation there is a simple distinction which can be drawn between monitoring being about routinely collected information and evaluation being about more ‘one-off’ studies. This distinction is useful sometimes, however it is not much good for any more sophisticated discussion of differences in evaluation, monitoring, performance management and related processes which people use in particular situations.
A more useful approach is set out in an article I have just written on Distinguishing Evaluation from Other Processes. The problems with trying to differentiate high-level terms are: 1) that we cannot control the way that stakeholders use them in the field; and, 2) these high-level terms for processes are not fully mutually exclusive, that is, sometimes the same activities are undertaken within processes even though the processes are thought to be distinct (e.g. evaluation, performance management, etc.). (more…)
Developing a comprehensive sector visual outcomes model January 20, 2009
Posted by Paul Duignan in : Communicating outcomes models, DoView, Easy Outcomes, Indicators, Outcomes models, Reporting systems, Uncategorized , 2comments
In earlier blog postings I’ve talked about the use of large scale outcomes models for various purposes such as overall policy development, evidence-based practice and monitoring and evaluation. The next version of DoView outcomes and evaluation software [[Disclosure: I am involved in the development of DoView] is going to allow images to be included in DoView files and in the web page models which can be created within DoView and then put up on the web. A mock-up of what parts of a visual sector plan using the upcoming version of DoView could look like is available. This will be a no cost update to DoView which is planned for release in February 2009.
What can be claimed about whether a program works or not from a logic model? January 13, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Impact evaluation, Outcomes models, Uncategorized , 1 comment so far
We sometimes hear things like: “a logic model was used to show that the program works”. I’m interested in tidying up such talk so that we are very clear about exactly what it is that is being claimed in regard to showing whether or not a program works. I’ve put up a new article in Knol which deals with the question of the types of claims we can make in regard to logic models (I call them outcomes models) and the types of arguments we can mount about whether or not programs work based on these claims. The article is here.The article is rather intense, but I think that it’s important that, as evaluators, we get on top of this sort of thing. In the article, I set out three claims that can be made in regard to logic models (or sub-parts of logic models). These are: (more…)
Top to bottom or left to right? Logic model conventions January 8, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Outcomes models, Using the approach , 4comments
There are various conventions for visualizing logic models (or outcomes models as I call them to include the wide range of different models evaluators work with – program logics, logic models, outcomes hierarchies, theories of change, program theories, strategy maps, ends-means diagrams, results chains etc.)
I have put up a knol article in the Outcomes Theory Knowledge Base which talks about why I think there are advantages to a top to bottom rather than a left to right approach to drawing logic models. Of course, if you are in an organization where you are told to draw left to right logic models, then go for it and some people who use DoView software use it to draw left to right models. The article is here. (more…)
Drawing logic models (outcomes models) January 5, 2009
Posted by Paul Duignan in : Communicating outcomes models, DoView, Outcomes models, Outcomes theory, Uncategorized , 1 comment so far
I have just finished a substantial article on what I call outcomes models. They go by names such as: logic models, program logics, program theories, theories of change, ends-means diagram, strategy maps etc. and are used widely through evaluation.
In the article I attempt to go beyond the traditional received conventions about logic models which I think are usually unexamined in ‘how to’ books and much of the discussion of logic models.
The essence of my argument is that:
- There are six possible purposes for outcomes models – the first being to provide a picture of ‘what it is believed causes what to happen’ – what I call a ‘full’ outcomes model.
- Because people attempt to also use models to indicate measurability and demonstrability (attribution of changes in high-level steps and outcomes to a program) they end up truncating their models so much (limiting them to the measurable or the demonstrable) that they fail to achieve the first purpose of creating a picture of a ‘full’ model. Measurement and demonstrability (attribution) should be done after the model has been built.
- Most ‘technologies of representation’ – tables, single page diagrams etc – are hopeless at representing anything like a ‘full’ model of the causality behind a program. Hence they attract attacks as inadequate representations of programs.
- Models should be drawn in logic model software such as DoView so as to be able to represent full models (as large as they need to be, anything linked to anything else) and then measurement and demonstrability (attribution) can be mapped back onto them. This means they can achieve the purposes of the traditional conventional approaches to logic models without having their limitations.
If you have a moment, check out the article, any feedback appreciated. The article is here.
Paul Duignan PhD
Ease of impact evaluation and bias within ‘what works’ evidence-based practice systems January 4, 2009
Posted by Paul Duignan in : Attribution, Evaluation planning, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a comment
Back on deck after Christmas and turning the mind back from eating and having a great time with relatives to outcomes and evaluation again. I’ve just put up an article in the Outcomes Theory Knowledge Base about the issue of evidence-based practice systems not taking into account variations in the ease of undertaking impact evaluation between program types.
There are many examples of evidence-based practice systems being set up (two classic examples are the Cochrane Collaboration in health care and the Campbell Collaboration in the social policy area) which are attempting to use information from impact evaluation (evaluation which demonstrates attribution of improvements in high level outcomes to particular programs). I think that such initiatives are generally a great idea. My article deals with a particular situation which sometimes arises which needs to be addressed in any well constructed ‘what works’ system. (more…)