A Systematic Outcomes Analysis framework for psychotherapy evaluation February 12, 2008
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes systems architecture, Systematic Outcomes Analysis, Outcomes models, Easy Outcomes, Evaluation planning, DoView , 2comments
In my last blog posting (which you should read before this one) I talked about using Systematic Outcomes Analysis to define the basic tasks one needs to do in quality assurance, monitoring and evaluation and how this can avoid the need for a protracted theoretical discussion about the difference between quality assurance and program evaluation. I was using the example of an illustrative Systematic Outcomes Analysis framework I set up based on an outcomes logic model in regard to psychotherapy which I’ve posted on the Outcomes Models site. Here’s the PDF of the DoView file. Using the Systematic Outcomes Analysis approach, indicators and evaluation questions are mapped onto the outcomes logic model (indicators are marked with a yellow icon and evaluation questions with a green circular icon). This blog posting looks in more detail at ways stakeholders can use such a framework once it’s been developed. (more…)
Avoiding the question: Defining quality assurance versus program evaluation February 12, 2008
Posted by Paul Duignan in : Using the approach, Doing evaluation more efficiently, Outcomes systems architecture, Systematic Outcomes Analysis, Outcomes models, DoView, Easy Outcomes, Uncategorized , 3comments
Sometimes it’s more useful to avoid initially answering a question that’s posed in a particular way because there’s a better way of addressing the concern that lies behind the question. Such is the case if you’re ever asked to define the difference between quality assurance (or monitoring) and program evaluation.
Seeing the question as a theoretical one and attempting to find a definition which works has some similarities to the situation where you’re building a house and someone keeps wanting you to stop and define, from a theoretical point of view, the difference between the kitchen and the dining room. Now, some people do stuff in the dining room that others do in the kitchen, and some do stuff in the kitchen that others do in the dining room. Still other people don’t really have any theoretical problems because they have a kitchen/dining area where they do both kitchen and dining room stuff.
A more fruitful way of working with the question of the difference between quality assurance (or monitoring) and program evaluation is to attempt to identify all of the stuff (tasks) that you would do under each of these. Once you’ve done that, you can then decide whether or not you need to spend a lot of time defining the difference between the two if everybody concerned is clear about which of the underlying tasks are, and are not, being done by whom. (more…)
Indicators, targets, benchmarks - sorting out the terminology November 27, 2007
Posted by Paul Duignan in : Outcomes systems architecture, Outcomes theory, Indicators, Measurement, Using the approach , add a commentMany different terms are used in the outcomes and performance management area for measurement and indicators. Often there is considerable confusion about these terms. The short definitions I use from outcomes theory are:
- Outcomes - causes or effects in the real world. Whether or not such causes can be measured is a separate issue (see previous blog for features of outcomes)
- Steps - lower level causes which lead to higher-level outcomes. Because causal processes reside in causal hierarchies, outcomes at one level can be steps for achieving even higher-level outcomes, therefore to refer to causes and effects at any level the general term ‘outcomes and steps’ is used.
- Measurements - measure whether or not an outcome or step has occurred (or how much of it has occurred).
- Indicator - a measurement of an outcome or step.
- Target - a level on an indicator.
- Benchmark - levels on indicators already achieved by other players, or by the same player at an earlier time or in another setting.
- Priorities - an outcome or step which is thought to be the most important for a player to focus their efforts on changing.
Reporting on outcomes to multiple bodies with different outcomes structures November 14, 2007
Posted by Paul Duignan in : Accountability, Outcomes systems architecture, Using the approach, Outcomes models, Easy Outcomes, DoView , add a commentIn a workshop the other day the issue arose of how you deal with the situation where you have to report to a number of different outside organizations on your outcomes. Now, this is not much of a problem where the outside organizations don’t make any implicit or explicit demands on how you report. The way to proceed in such cases is to simply develop your outcomes model and report back to them on it.
However, with more and more organizations thinking in terms of outcomes, they are starting to have outcomes structures of various types themselves. (more…)
Cutting out parallel processes - the CEO’s responsibility October 11, 2007
Posted by Paul Duignan in : Outcomes systems architecture, Outcomes models, Easy Outcomes , add a comment
Outcomes systems are a CEO and other managers’ way of aligning what happens in an organization with the outcomes it’s trying to achieve. It’s rather like the Captain on the bridge of a large aircraft carrier needing a well functioning communications system which flows all the way throughout the vessel. If they don’t have it, they’re not going to get the carrier to go where they want it to. The outcomes systems in many organizations are like an aircraft carrier which has a confusing mess of communications systems - all sorts of flags flying from the bridge, a telephone system, several radio systems, people shouting, small notes passed to individual sailors etc. All these have been introduced at different times, some are badly designed, some are now not functioning in the way they were originally designed to. Meanwhile, the captain is running around the bridge yelling into phones, scribbling on scraps of paper and waving flags while the carrier meanders around the ocean turning in larger and larger circles. (more…)