Standardized visual evaluation plans - quick and effective October 9, 2008
Posted by Paul Duignan in : Using the approach, Communicating outcomes models, Doing evaluation more efficiently, Systematic Outcomes Analysis, Outcomes models, DoView, Easy Outcomes, Evaluation planning, Uncategorized , add a comment
I’ve not had much time to blog recently due to building a number of large outcomes models for public sector organizations; having input into the further development of DoView; and presenting at international evaluation conferences on Easy Outcomes, DoView and related evaluation and outcomes topics. A lot has been happening though, from version 1.14, DoView is now able to create web page versions of its visual outcomes models. I’ll do several postings showing how this new feature can be used. The first is that now, once an outcomes model has been built in DoView, the user can quickly create a web page version of the same model and then have it put up on an intranet or the internet. You can see (and use) a number of examples at OutcomesModels.org. The second great thing is that you can now produce visual evaluation plans will save you a great deal of time. I delivered a paper on this at the recent European Evaluation Society Biennial Conference in Lisbon. (more…)
Avoiding the question: Defining quality assurance versus program evaluation February 12, 2008
Posted by Paul Duignan in : Using the approach, Doing evaluation more efficiently, Outcomes systems architecture, Systematic Outcomes Analysis, Outcomes models, DoView, Easy Outcomes, Uncategorized , 3comments
Sometimes it’s more useful to avoid initially answering a question that’s posed in a particular way because there’s a better way of addressing the concern that lies behind the question. Such is the case if you’re ever asked to define the difference between quality assurance (or monitoring) and program evaluation.
Seeing the question as a theoretical one and attempting to find a definition which works has some similarities to the situation where you’re building a house and someone keeps wanting you to stop and define, from a theoretical point of view, the difference between the kitchen and the dining room. Now, some people do stuff in the dining room that others do in the kitchen, and some do stuff in the kitchen that others do in the dining room. Still other people don’t really have any theoretical problems because they have a kitchen/dining area where they do both kitchen and dining room stuff.
A more fruitful way of working with the question of the difference between quality assurance (or monitoring) and program evaluation is to attempt to identify all of the stuff (tasks) that you would do under each of these. Once you’ve done that, you can then decide whether or not you need to spend a lot of time defining the difference between the two if everybody concerned is clear about which of the underlying tasks are, and are not, being done by whom. (more…)
Indicators, targets, benchmarks - sorting out the terminology November 27, 2007
Posted by Paul Duignan in : Outcomes systems architecture, Outcomes theory, Indicators, Measurement, Using the approach , add a commentMany different terms are used in the outcomes and performance management area for measurement and indicators. Often there is considerable confusion about these terms. The short definitions I use from outcomes theory are:
- Outcomes - causes or effects in the real world. Whether or not such causes can be measured is a separate issue (see previous blog for features of outcomes)
- Steps - lower level causes which lead to higher-level outcomes. Because causal processes reside in causal hierarchies, outcomes at one level can be steps for achieving even higher-level outcomes, therefore to refer to causes and effects at any level the general term ‘outcomes and steps’ is used.
- Measurements - measure whether or not an outcome or step has occurred (or how much of it has occurred).
- Indicator - a measurement of an outcome or step.
- Target - a level on an indicator.
- Benchmark - levels on indicators already achieved by other players, or by the same player at an earlier time or in another setting.
- Priorities - an outcome or step which is thought to be the most important for a player to focus their efforts on changing.
Reporting on outcomes to multiple bodies with different outcomes structures November 14, 2007
Posted by Paul Duignan in : Accountability, Outcomes systems architecture, Using the approach, Outcomes models, Easy Outcomes, DoView , add a commentIn a workshop the other day the issue arose of how you deal with the situation where you have to report to a number of different outside organizations on your outcomes. Now, this is not much of a problem where the outside organizations don’t make any implicit or explicit demands on how you report. The way to proceed in such cases is to simply develop your outcomes model and report back to them on it.
However, with more and more organizations thinking in terms of outcomes, they are starting to have outcomes structures of various types themselves. (more…)
New outcomes models (program logics) put up on outcomesmodels.org October 17, 2007
Posted by Paul Duignan in : Using the approach, Outcomes models, DoView , add a comment
We’ve just put a number of new outcomes models (program logics) up on the outcomesmodels.org site. We’re planning to continue adding to the outcomes model collection at outcomesmodels.org for two reasons. The first is to give examples of how models can be drawn in DoView. The more examples people see, the easier they find it becomes to build DoView models. The second is to allow people to grab either the whole of one of these models or just some of the slices from one or more models to use as a start in building their own models. DoView’s unique modular approach using multiple diagram slices makes it very easy to do this. (more…)
Beware lumpers, splitters and slice globblers when you’re building outcomes models October 15, 2007
Posted by Paul Duignan in : Using the approach, Standards, Outcomes models, DoView , add a comment
When you’re drawing outcomes models (program logics, strategy maps, means-ends diagrams, results chains etc), using DoView or other software there’re a few things which will come up on a regular basis. The first of these is the personality difference between lumpers and splitters who are present in the room. Lumpers, obviously want to lump and splitters, well, they just want to split. So if you’re working on an outcomes model which at a high level includes, say, social and economic outcomes, a lumper will want a single outcome which goes ‘complete social and economic whatever’. A splitter, on the other hand, will want to have two separate outcomes - a ’social whatever’ and an ‘economic whatever’ outcome. Neither of them is right or wrong, although in the first instance I often let the splitter have their way because outcomes or steps can always be combined at a later stage if needed. (more…)
Get out of the dark - cut outcomes confusion with the five features of outcomes October 14, 2007
Posted by Paul Duignan in : Outcomes theory, Using the approach, Outcomes models , 3comments
Many terms are used in outcomes systems to describe the elements which go into such systems representing the steps in the causal processes leading from low level activities through to final high-level outcomes. Terms used include: outcomes, intermediate outcomes, outputs, activities, key drivers, key priorities etc. These terms tend to appear in outcomes models (program logics, strategy maps, ends-means diagrams, results chains etc.). In models which are drawn vertically, they normally appear down the left-hand side. There’s often confusion as to the exact distinctions between these terms and many a happy hour is spent by those trying to implement outcomes systems struggling to work out what a particular step should be called and exactly where it should go in an outcomes model.
Outcomes theory can help clarify this issue by using the five major features of outcomes identified in the theory. Outcomes or the steps which lead to them can be: influencable, controllable, measurable, attributable and accountable. Thinking about it in this way can save you a lot of time and confusion when working with outcomes systems. (more…)
Avoid being an outcomes model ‘Go-Between’ September 26, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, Using the approach, Standards, Outcomes models, DoView , add a comment
A while ago a colleague recounted to me how they’d ended up pulling out their hair because they found themselves in a ‘Go-Between’ role when drawing an outcomes model (also called program logics, results chains, strategy maps, ends-means diagrams). You need to try to avoid this at all costs, although when dealing with high level stakeholders it’s often not easy to do so. I found myself in this role on a major project a while ago and I certainly didn’t enjoy it.
(more…)