Why just about every indicator system in the world needs to be fixed! April 5, 2009
Posted by Paul Duignan in : Accountability, Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Standards, Using the approach , 1 comment so farI’ve just posted a new article in the Outcomes Theory Knowledge Base on why it is essential to map indicators onto an underlying visual outcomes model. I blogged a little while ago about why we should be wary of too-tidy indicator sets and in the article I explain why. The vast majority of indicator systems in the world suffer from the problem set out in the article – they are just a straight list of indicators set out in tabular format. They give the user no idea as to whether a number of important steps and outcomes are not being measured. Those using such systems remain blissfully unaware of this. In my view, all these straight indicator sets need to be fixed. It’s not particularly difficult, it just requires some work. How to draw the underlying outcomes models is set out in the outcomes model standards and how to then use such models for indicator mapping and many other things is described in detail in the applied version of outcomes theory – Easy Outcomes. (more…)
Using visual outcomes models to describe and communicate best practice April 3, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Research influening policy, Strategic planning, Using the approach , 4commentsYesterday I blogged about what is meant by the term ‘best practice’. As I said then, I think that there is some conceptual confusion out there about it, and I am not sure that I have yet tidied it up my own thinking about it in a satisfactory way. However, the great thing is that regardless of how we define it, the idea of identifying the types of things that people currently call best practice and communicating these between programs is a great idea. The most difficult thing in many cases is to get best practice actually applied to on the ground after we have identified it. I have put up an Outcomes Theory Knowledge Base article (link to the article is at the bottom of this posting) on the issue suggesting that an efficient way of describing and communicating best practice may be to use visual outcomes models (a type of logic model). (more…)
Ease of impact evaluation and bias within ‘what works’ evidence-based practice systems January 4, 2009
Posted by Paul Duignan in : Attribution, Evaluation planning, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a comment
Back on deck after Christmas and turning the mind back from eating and having a great time with relatives to outcomes and evaluation again. I’ve just put up an article in the Outcomes Theory Knowledge Base about the issue of evidence-based practice systems not taking into account variations in the ease of undertaking impact evaluation between program types.
There are many examples of evidence-based practice systems being set up (two classic examples are the Cochrane Collaboration in health care and the Campbell Collaboration in the social policy area) which are attempting to use information from impact evaluation (evaluation which demonstrates attribution of improvements in high level outcomes to particular programs). I think that such initiatives are generally a great idea. My article deals with a particular situation which sometimes arises which needs to be addressed in any well constructed ‘what works’ system. (more…)
Evaluation Advisory Committees – getting it right at the start December 16, 2008
Posted by Paul Duignan in : Doing evaluation more efficiently, Evaluation planning, Using the approach , 1 comment so far
Recently I attended the first meeting of an Evaluation Advisory Committee and I was reminded once again of the importance of setting up evaluations properly at the start. I have set out below the issues which need to be dealt with when such committees are set up. This list is drawn from my experience on a number of these types of committees. The issues you need to deal with include: (more…)
Impact evaluation’s day in the sun (Part III) December 3, 2008
Posted by Paul Duignan in : Attribution, Evaluation debates, Evaluation planning, Impact evaluation, Outcomes systems architecture , 1 comment so far
[Please Read Part I and Part II first]. Now that I have got to Part III of this posting on the impact evaluation debate, the time has come for me to front up and tell you how I think things should be done if I were in charge of the world. First, the concept of there being a shoot-out between impact evaluation and other types of evaluation is about as sensible as trying to have a debate as to whether a fork or a knife is the better kitchen utensil. It obviously depends on what you want to do with it. This does not, of course mean that it might not be reasonable to argue that there are not enough knives in the kitchen at a particular point in time and that we should go out and encourage knife collecting rather than just getting more and more forks. (more…)
Impact evaluation’s day in the sun (Part I) November 30, 2008
Posted by Paul Duignan in : Attribution, Evaluation debates, Evaluation planning, Impact evaluation , 2comments
When you’ve been around for a while you see things come and go. Impact evaluation – evaluation looking at the longer-term (or ‘higher-level’ outcomes within an visual outcomes model) is currently enjoying a resurgence (sometimes its just called plain outcome evaluation). I’ve been attending a number of international evaluation conferences recently and had the pleasure of going to several workshops from experts on impact evaluation – Gary Henry from the University of North Carolina who ran a workshop at the United Kingdom Evaluation conference in Bristol and Charles Reichardt from the University of Denver ran a workshop on estimating the effects of interventions at the American Evaluation Conference in Denver. I thought both were excellent and showed the level of sophistication some of the experts thinking about impact/outcome evaluation are going to to do the job properly. I will blog in more detail about what they were saying a little later. However, these presentations were little oases of calm in a wider and more heated debate in evaluation circles about the resurgence of impact evaluation. (more…)
Standardized visual evaluation plans – quick and effective October 9, 2008
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Systematic Outcomes Analysis, Uncategorized, Using the approach , add a comment
I’ve not had much time to blog recently due to building a number of large outcomes models for public sector organizations; having input into the further development of DoView; and presenting at international evaluation conferences on Easy Outcomes, DoView and related evaluation and outcomes topics. A lot has been happening though, from version 1.14, DoView is now able to create web page versions of its visual outcomes models. I’ll do several postings showing how this new feature can be used. The first is that now, once an outcomes model has been built in DoView, the user can quickly create a web page version of the same model and then have it put up on an intranet or the internet. You can see (and use) a number of examples at OutcomesModels.org. The second great thing is that you can now produce visual evaluation plans will save you a great deal of time. I delivered a paper on this at the recent European Evaluation Society Biennial Conference in Lisbon. (more…)
A Systematic Outcomes Analysis framework for psychotherapy evaluation February 12, 2008
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Outcomes systems architecture, Systematic Outcomes Analysis , 2comments
In my last blog posting (which you should read before this one) I talked about using Systematic Outcomes Analysis to define the basic tasks one needs to do in quality assurance, monitoring and evaluation and how this can avoid the need for a protracted theoretical discussion about the difference between quality assurance and program evaluation. I was using the example of an illustrative Systematic Outcomes Analysis framework I set up based on an outcomes logic model in regard to psychotherapy which I’ve posted on the Outcomes Models site. Here’s the PDF of the DoView file. Using the Systematic Outcomes Analysis approach, indicators and evaluation questions are mapped onto the outcomes logic model (indicators are marked with a yellow icon and evaluation questions with a green circular icon). This blog posting looks in more detail at ways stakeholders can use such a framework once it’s been developed. (more…)
What’s the Easy Outcomes site all about? September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning , add a commentI’ve realized that I’ve not put up a post about the Easy Outcomes site even though it’s been up for a few weeks. Easy Outcomes takes the Systematic Outcomes Analysis approach and applies it using DoView outcomes software. We developed it in response to feedback that the Systematic Outcomes Analysis site is a bit intense for those who just want to work out how to develop a good evaluation plan, but don’t want to get buried in the detail of the approach too soon. (more…)
New Systematic Outcomes Analysis case study – evaluating an academic research rating scheme September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Evaluation planning, Systematic Outcomes Analysis , 1 comment so farI’ve recently put up the link to another case study on the Systematic Outcomes Analysis site. This is a report detailing an evaluation and monitoring plan for the Performance Based Research Fund (PBRF). The PBRF is a national academic research output assessment system and the monitoring and evaluation framework was produced for the government organization responsible for the tertiary sector – the Tertiary Education Commission.
The report is a good example of a comprehensive use of Systematic Outcomes Analysis (in the report the approach went by one of its earlier names – REMLogic). One of the interesting things about the PBRF is that it’s an evaluation system in its own right and developing an evaluation framework for an evaluation system presents an interesting technical challenge. I’ll put up a blog post about that challenge some time in the future. (more…)