Why just about every indicator system in the world needs to be fixed! April 5, 2009
Posted by Paul Duignan in : Accountability, Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Standards, Using the approach , 1 comment so farI’ve just posted a new article in the Outcomes Theory Knowledge Base on why it is essential to map indicators onto an underlying visual outcomes model. I blogged a little while ago about why we should be wary of too-tidy indicator sets and in the article I explain why. The vast majority of indicator systems in the world suffer from the problem set out in the article – they are just a straight list of indicators set out in tabular format. They give the user no idea as to whether a number of important steps and outcomes are not being measured. Those using such systems remain blissfully unaware of this. In my view, all these straight indicator sets need to be fixed. It’s not particularly difficult, it just requires some work. How to draw the underlying outcomes models is set out in the outcomes model standards and how to then use such models for indicator mapping and many other things is described in detail in the applied version of outcomes theory – Easy Outcomes. (more…)
Using visual outcomes models to describe and communicate best practice April 3, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Research influening policy, Strategic planning, Using the approach , 4commentsYesterday I blogged about what is meant by the term ‘best practice’. As I said then, I think that there is some conceptual confusion out there about it, and I am not sure that I have yet tidied it up my own thinking about it in a satisfactory way. However, the great thing is that regardless of how we define it, the idea of identifying the types of things that people currently call best practice and communicating these between programs is a great idea. The most difficult thing in many cases is to get best practice actually applied to on the ground after we have identified it. I have put up an Outcomes Theory Knowledge Base article (link to the article is at the bottom of this posting) on the issue suggesting that an efficient way of describing and communicating best practice may be to use visual outcomes models (a type of logic model). (more…)
Simplifying outcomes terminology – angels dancing on the head of a pin! April 1, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes theory, Reporting systems, Using the approach , add a commentIn the middle ages, churchmen occupied themselves by having long discussions sitting around the fire drinking fine wine and discussing how many angels could dance on the head of a pin. It was tough, but someone had to do it! That was until someone called William of Occam came along. He is famous for spoiling the party with something called ‘Occam’s Razor’ – one way of describing the way he operated was that the did not want people to ‘multiply entities beyond necessity’ (see Wikipedia entry). What he was talking about was trying to keep things as simple as possible. This is the approach which is taken within outcomes theory. (more…)
Lines, arrows and ‘Engineering’ solutions March 20, 2009
Posted by Paul Duignan in : Communicating outcomes models, DoView, Easy Outcomes, Using the approach , add a commentAt a national philanthropy conference I presented at this week I demonstrated using the Easy Outcomes approach as a way for grantees to communicate what they are doing in their programs to their philanthropic funders. The visual models underlying the Easy Outcomes approach are drawn in DoView outcomes and evaluation software. The particular example I used was a user-friendly mock-up which I will put up on the internet in a week or so. It uses the latest version of DoView (1.17) which is planned for release in a couple of weeks – it is very cool in that it allows images to be inserted into a DoView file – really bringing outcomes models (logic models) to life. Check out this initial mock-up here (it will also let you drill-down with a hyperlink ‘hop-to’ beneath text, indicators, evaluation questions etc.). There were lots of positive comments about the model, including one community group woman saying ”Hallelujah, finally someone has started talking our language and realzing that we like working with pictures and images not just tables and text’. However, one other person did comment that they though the model looked rather like an ‘engineering approach’. (more…)
Tables versus visual models March 19, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Outcomes models, Reporting systems, Using the approach , 3commentsEach day this week I am blogging on some of the themes that came up at a national philanthropy conference earlier in the week where I presented on the Easy Outcomes approach. There was some discussion in the conference session about the dynamic nature of what grantees do and how it is hard to capture such dynamism in any rigid system of evaluation. However, as is often the case, at the end of the day, some tabular logic models were presented as an example of how to set out an evaluation. I am not criticizing their presentation, because this is currently standard practice in evaluation planning, and the use of some sort of logic model is always much better than not having one. However, in my view, we fall back on the use of tables because we do not realize that there are better ‘technologies of representation’ (i.e. visual outcomes models) that we could use.
(more…)
Beware of suspiciously tidy indicator sets March 19, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Easy Outcomes, Indicators, Measurement, Outcomes models, Reporting systems , 1 comment so farI’ve just come away from presenting at a national Philanthropy conference as part of a half day session on evaluation and outcomes. I was presenting on the use of the Easy Outcomes approach as a way of grantees structuring outcomes, indicators and evaluation. I will tidy up the outcomes model I used and post a link to it in a blog in a week or so. A lot of interesting points came up in the discussion and I will blog on several of them over the next few days. The first one is to be beware of suspiciously tidy indicator sets. The Easy Outcomes approach gets people to draw an outcomes model (intervention logic) of what they are trying to do without worrying about what they can and can’t measure and what they can, and can’t demonstrate is attributable to their particular project (both of these issues are dealt with later in the process). You draw the models using the guidelines here.
(more…)
What’s in a definition? March 15, 2009
Posted by Paul Duignan in : Blog info, Communicating outcomes models, Outcomes systems architecture, Outcomes theory , 2commentsI have not blogged for a number of weeks due to family illness and a bereavement. However, now I am back on deck and planning to increase the frequency of my blogs – shorter on some occassions and more often, but there will still be some substatial ones for those who want substance. If there are any issues you want me to blog on, or any questions you would like me to answer in future blogs, just post a comment to this posting.First up, I have now added to the Outcomes Theory Knowledge Base a set of definitions as they are used in outcomes theory. (more…)
Developing a comprehensive sector visual outcomes model January 20, 2009
Posted by Paul Duignan in : Communicating outcomes models, DoView, Easy Outcomes, Indicators, Outcomes models, Reporting systems, Uncategorized , 2comments
In earlier blog postings I’ve talked about the use of large scale outcomes models for various purposes such as overall policy development, evidence-based practice and monitoring and evaluation. The next version of DoView outcomes and evaluation software [[Disclosure: I am involved in the development of DoView] is going to allow images to be included in DoView files and in the web page models which can be created within DoView and then put up on the web. A mock-up of what parts of a visual sector plan using the upcoming version of DoView could look like is available. This will be a no cost update to DoView which is planned for release in February 2009.
What can be claimed about whether a program works or not from a logic model? January 13, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Impact evaluation, Outcomes models, Uncategorized , 1 comment so far
We sometimes hear things like: “a logic model was used to show that the program works”. I’m interested in tidying up such talk so that we are very clear about exactly what it is that is being claimed in regard to showing whether or not a program works. I’ve put up a new article in Knol which deals with the question of the types of claims we can make in regard to logic models (I call them outcomes models) and the types of arguments we can mount about whether or not programs work based on these claims. The article is here.The article is rather intense, but I think that it’s important that, as evaluators, we get on top of this sort of thing. In the article, I set out three claims that can be made in regard to logic models (or sub-parts of logic models). These are: (more…)
Top to bottom or left to right? Logic model conventions January 8, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Outcomes models, Using the approach , 4comments
There are various conventions for visualizing logic models (or outcomes models as I call them to include the wide range of different models evaluators work with – program logics, logic models, outcomes hierarchies, theories of change, program theories, strategy maps, ends-means diagrams, results chains etc.)
I have put up a knol article in the Outcomes Theory Knowledge Base which talks about why I think there are advantages to a top to bottom rather than a left to right approach to drawing logic models. Of course, if you are in an organization where you are told to draw left to right logic models, then go for it and some people who use DoView software use it to draw left to right models. The article is here. (more…)