Avoiding the question: Defining quality assurance versus program evaluation February 12, 2008
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes models, Outcomes systems architecture, Systematic Outcomes Analysis, Uncategorized, Using the approach , trackback
Sometimes it’s more useful to avoid initially answering a question that’s posed in a particular way because there’s a better way of addressing the concern that lies behind the question. Such is the case if you’re ever asked to define the difference between quality assurance (or monitoring) and program evaluation.
Seeing the question as a theoretical one and attempting to find a definition which works has some similarities to the situation where you’re building a house and someone keeps wanting you to stop and define, from a theoretical point of view, the difference between the kitchen and the dining room. Now, some people do stuff in the dining room that others do in the kitchen, and some do stuff in the kitchen that others do in the dining room. Still other people don’t really have any theoretical problems because they have a kitchen/dining area where they do both kitchen and dining room stuff.
A more fruitful way of working with the question of the difference between quality assurance (or monitoring) and program evaluation is to attempt to identify all of the stuff (tasks) that you would do under each of these. Once you’ve done that, you can then decide whether or not you need to spend a lot of time defining the difference between the two if everybody concerned is clear about which of the underlying tasks are, and are not, being done by whom.
To identify these underlying tasks we need a framework for thinking about the tasks which lie beneath quality assurance, monitoring and evaluation systems. Systematic Outcomes Analysis and its user-friendly companion approach – Easy Outcomes provides such a framework. In this approach, there are five tasks which need to be addressed by any outcomes system (outcomes systems are any quality assurance, monitoring, evaluation or related system). Each of these tasks are dealt with by one of five corresponding building-blocks which should lie beneath any well-functioning outcomes system (the building blocks are set out in the OIE Basic Diagram which is the picture at the top of this posting). The tasks are:
- Working out how it’s believed the program is meant to work- this is dealt with by making explicit the logic model (outcomes model) of the program. This sets out all of the lower-level steps which are needed to achieve high-level outcomes. Such models can be in someone’s head, on the back of an envelope, in a narrative document, in a mathematical model, or in a drawn model (or in all of these!). However, regardless of its form it should be explicit not implicit.
- Routinely measuring indicators of progress toward achieving steps and outcomes – for any outcomes model, indicators may or may not be able to be identified for each of the steps and outcomes within a model. Tracking those that can be measured, lets us find out whether we are on course to achieving the steps and outcomes we want. However, just measuring the progress of such indicators, while important, does not necessarily say anything about why they are improving (or not improving). Therefore such indicators are called not-necessarily attributable indicators. This means that their mere measurement is not taken as proof that changes in them are being caused by a particular program.
- Identifying those routine indicators which can be attributed to a particular program – it’s useful, where possible, to be able to attribute change in indicators for steps or outcomes to a particular program, this may or may not be possible. For some indicators, the mere fact that they change is accepted as evidence that such change can be attributed to a particular program. These tend to indicators for lower-level steps and the traditional measurement of outputs (the goods and services produced by a program) are examples of such attributable indicators.
- Doing analysis or experimentation to attribute change in high-level outcomes to a program- where routinely collected attributable indicators are not available for high-level outcomes, the attempt is sometimes made to use analysis or experimentation to answer one or more high-level outcome evaluation questions. High-level outcome evaluation designs are used for this and Systematic Outcomes Analysis identifies seven possible types of such designs which may or may not be appropriate, feasible and affordable in the case of a particular program. (More information on these designs here).
- Describing and improving the outcomes model - another set of evaluation questions, non-outcome evaluation (formative and process evaluation) questions, are used to describe the course and context of a program (process evaluation) and, in the case of formative evaluation, to improve the implementation of the outcomes logic model in a particular case.
In response to a recent posting on an evaluation list asking about the difference between quality assurance and program evaluation in the case of psychotherapy, I’ve put a simple psychotherapy outcomes model up on the Outcomes Models site (this is a repository of free to use outcomes logic models in DoView outcomes software and PDF file format). Have a look at the PDF of the psychotherapy outcomes model here.
I’ve taken the model and, using the five building-block framework set out above, I’ve put some indicators (represented by yellow icons) and some evaluation questions (represented by green circle icons) onto the model. Here’s a screenshot of the relevant diagram (slice) from the model.
Once you frame the problem up in this way, stakeholders can have a coherent discussion about any aspect of quality assurance (monitoring) or evaluation. The discussion is now in terms of which tasks (e.g. examining the outcomes model, reporting on indicators or answering evaluation questions) are, or should, be being done and who should do them.
Using this approach, the whole question of what is quality assurance and what is program evaluation no longer has any urgency because stakeholders realize that they’ve identified the underlying tasks which need to be done and are working away at doing these. If you like, you can group some some of the tasks, typically the routine measurement of indicators, and call that quality assurance. If you also want to, you can group the answering the evaluation questions and call that program evaluation. Alternatively, you could group the tasks in various other ways and call them whatever you like, it does not really matter much if you are always using the underlying framework presented here to make sure that you’re getting done the tasks you want to get done.
In my next blog posting I’ll discuss in more detail how stakeholders can use a framework like this.
Paul Duignan (outcomesblog.org)
Comments»
[...] my last blog posting I talked about using Systematic Outcomes Analysis to define the basic tasks one needs to do in [...]
The Easy Outcomes framework provides structured, detailed, and thoughtful ways to evaluate effectiveness of the operation of any program. I like the way that the model is divided into 10 simple steps to guide evaluators to comple a thorough evaluation for any program. In addition, I really like the way evaluation questions are included into the outcomes model to help evaluators and stakeholders focus on specific aspects of the program that need to be evaluated. The questions are also helpful in guiding stakeholders make appropriate and positive changes to improve their program without making unnecessary changes to aspects of the program that are operating well.
The easy outcomes model has a lot to offer to evaluation experts as well as to internal evaluators who are not experts in the evaluation process. Following the ten easy steps, internal evaluators can easily find ways to discuss and assess their programs in the objective and scentific light.
Systematic outcome anaysis provides user-friendly and structured steps for conducting outcome evaluation. It offers a simple and understandable framework for a novice evaluator to follow and to begin evaluation process.
[...] have blogged before about what I see as the wrong way to approach the problem of differentiating evaluation from [...]