New Systematic Outcomes Analysis case study – evaluating an academic research rating scheme September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Evaluation planning, Systematic Outcomes Analysis , trackbackI’ve recently put up the link to another case study on the Systematic Outcomes Analysis site. This is a report detailing an evaluation and monitoring plan for the Performance Based Research Fund (PBRF). The PBRF is a national academic research output assessment system and the monitoring and evaluation framework was produced for the government organization responsible for the tertiary sector – the Tertiary Education Commission.
The report is a good example of a comprehensive use of Systematic Outcomes Analysis (in the report the approach went by one of its earlier names – REMLogic). One of the interesting things about the PBRF is that it’s an evaluation system in its own right and developing an evaluation framework for an evaluation system presents an interesting technical challenge. I’ll put up a blog post about that challenge some time in the future.
Working on this report was an interesting milestone for me because it led to me to thinking that there had to be a more efficient way of setting out Systematic Outcome Analysis monitoring and evaluation plans. Just using normal drawing and wordprocessing software was too cumbersome.
Systematic Outcomes Analysis puts the outcomes model (program logic, strategy map, ends-means diagram) at the center of your evaluation and monitoring plan. The idea is to map all indicators and evaluation questions back onto your outcomes model. The practical problem when using normal software is that you need to use a numbering system to number all outcomes and to link them to indicators and evaluation questions which you set out in tables. If you just had to do this once it would be considerable work. The problem gets worse however when you’re developing an evaluation and monitoring plan in consultation with various stakeholders which almost always is the case – unless it is a purely academic exercise. When working with changing drafts, the outcomes, indicator and evaluation question numbering constantly changes as outcomes, indicators or evaluation questions are added, deleted or combined.
It was the experience of working on this project which led me to the conclusion that there had to be an easier way. We developed DoView outcomes software as a result of wanting to find a more efficient way of both building outcomes models and organizing and amending a monitoring and evaluation plan. With it you can put indicators and evaluation questions right onto your outcomes model and by using ‘clones’ (live copies), if you update an outcome, indicator or evaluation questions anywhere in the model it automatically will change right across the model.
Paul Duignan (outcomesblog.org)
Comments»
[...] Based Research Fund (PBRF) outcomes model from the evaluation plan discussed in an earlier blog posting. Here the program was all about measuring the quality of academics’ research output and so [...]