Distinguishing Evaluation from Monitoring (and other processes such as Performance Management and Assessment) January 30, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Systematic Outcomes Analysis, Using the approach , trackbackI have blogged before about what I see as the wrong way to approach the problem of differentiating evaluation from related activities (e.g. monitoring, performance management, assessment). Last time it was about differentiating evaluation from quality assurance. The wrong way is to spend a lot of time on attempting to distinguish between processes by using high-level terms such as evaluation, performance management etc. A lot of ink and talk has been spilled on trying to do that over the years. In the case of monitoring and evaluation there is a simple distinction which can be drawn between monitoring being about routinely collected information and evaluation being about more ‘one-off’ studies. This distinction is useful sometimes, however it is not much good for any more sophisticated discussion of differences in evaluation, monitoring, performance management and related processes which people use in particular situations.
A more useful approach is set out in an article I have just written on Distinguishing Evaluation from Other Processes. The problems with trying to differentiate high-level terms are: 1) that we cannot control the way that stakeholders use them in the field; and, 2) these high-level terms for processes are not fully mutually exclusive, that is, sometimes the same activities are undertaken within processes even though the processes are thought to be distinct (e.g. evaluation, performance management, etc.).
The way I think that we need to approach the problem is to identify the set of activities which are included within evaluation, monitoring and related processes and use such a list of activities to specify exactly what is being done in a particular case regardless of what high-level term stakeholders use to describe that process.
We can use the Five Building-Blocks of Outcomes Systems Model (from this article here) from outcomes theory to specify, from a technical point of view, the activities which are being undertaken under any high-level term such as evaluation or performance management.
The The Five Building-Blocks model identifies five basic components of any outcomes system. Outcomes systems are evaluation, monitoring, performance management and related systems. These five building-blocks are:
1) an outcomes model (logic model);
2) not-necessarily demonstrably attributable indicators;
3) demonstrably attributable indicators;
4) attempts to establish attribution where demonstrably attributable indicators do not reach to the top of the outcomes model (impact/outcomes evaluation);
5) attempts to describe the outcomes model and its context as implemented in a particular situation (process evaluation) and attempts to improvement the implementation of an outcomes model in a particular situation (formative evaluation).
Using this approach we can specify any process or system for evaluating, measuring outcomes or performance by which of the five building-blocks it is using. This approach lets us escape from the endless attempts to attempt to establish once and for all a distinction between different processes such as evaluation, monitoring, performance management, assessment etc and our futile attempts (even if we did settle on tight definitions) to control the way stakeholders use these terms in the field.
Paul Duignan, PhD
Comments»
I think that additional terms that need to be clarified are performance measurement, then performance management. Quality assurance is not under the control of the evaluator or even the manager — because of stakeholder influence. Performance mesurement enables output, outcomes disinctions. – B
Barry, if you have a moment could you elaborate on this comment. How would you distinguish performance measurement and performance management? And how does the use of performance measurement enable the output/outcomes distinction.
I am from the “old school” of structure process and outcomes. That carries with it a concern about linking processes with outcomes. Until you understand – processes and outcomes, you cannot manage them. In health, many say now that the move is clearly away from manged care to performance measurement then to the management of performance. If you can measure price -quality relationships you can then manage them. I think performance assessment and measurement are the same thing. B
I meant to add that for me the different between output and outcomes is that the first is the product of the processes where outcomes are evrinomental impact assessments. In hospitals, DRGs are good examples of outputs of merging clinical and administrative processes. The outcome assessments of these processes is certainly another measure. B
I have to add that in my view the most misuderstood term in all of this is evaluation. I think that there is too little discussion in the knowledge base about the actual components of a system. If the system consists of inputs-throughput processes- resulting in outputs, then outcomes are the impact of the outputs on whatever the relevant environment is. The term evaluation could refer to the specification of any of the relationships that might take place across those sytem components – B