Standardized visual evaluation plans – quick and effective October 9, 2008
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Systematic Outcomes Analysis, Uncategorized, Using the approach , trackback
I’ve not had much time to blog recently due to building a number of large outcomes models for public sector organizations; having input into the further development of DoView; and presenting at international evaluation conferences on Easy Outcomes, DoView and related evaluation and outcomes topics. A lot has been happening though, from version 1.14, DoView is now able to create web page versions of its visual outcomes models. I’ll do several postings showing how this new feature can be used. The first is that now, once an outcomes model has been built in DoView, the user can quickly create a web page version of the same model and then have it put up on an intranet or the internet. You can see (and use) a number of examples at OutcomesModels.org. The second great thing is that you can now produce visual evaluation plans will save you a great deal of time. I delivered a paper on this at the recent European Evaluation Society Biennial Conference in Lisbon. Over the years of teaching evaluation planning, writing evaluation plans and peer reviewing evaluation plans, it has always struck me as strange that each evaluation plan is approached as if it were a separate exercise. Other disciplines have highly standardized ways of doing things, but for some reason evaluation has taken more of a ‘craft’ approach where each evaluation may be structured differently. This makes it hard for funders and other stakeholders wanting to quickly understand what it is that an evaluation is going to do. This is partially the result of the different schools of evaluation not being able to agree on a standard format for an evaluation plan which they believe will let them do justice to their way of doing things.
The Easy Outcomes approach was designed to provide a standardized framework for developing an evaluation plan which would allow any evaluator of any theoretical orientation to clearly set out what they are planning to do. This built on my earlier work with the Systematic Outcomes Analysis approach which attempted to do the same thing. For lack of appropriate visualization software, Systematic Outcomes Analysis tended to be cumbersome to apply in practice (e.g. see the PBRF evaluation plan which was set out in traditional textual narrative format – rather intense!).The concept behind the development of DoView was to allow for a visualization environment in which a user can build visual outcomes models (intervention logics, program logics) of any size.
The secondary objective was to allow such visual models to be used as the heart of a visual evaluation plan (or other types of plans).Now that these plans can also be produced as web page version, we can achieve significant time-saving in the production of evaluation plans and evaluation implementation. Here is an example of a web page version of a visual evaluation plan created in DoView. The logo of any organization can be included in the plan and note that the original DoView file can be included with the web page model so that anyone who has DoView can download the original file and edit it themselves. This visual approach to evaluation planning can save a great deal of time.
In the past, what evaluators tended to do was to develop a narrative evaluation plan as a long text document (say 50 to 100 pages). This was used by high-level stakeholders to make decisions about what evaluation they want undertaken. The reality of course is that most of them were too busy to read the document which had been so carefully written and polished by evaluation staff (as such documents need to be). Even though many of they do read the executive summary of such reports, it is difficult to convey the nuances of evaluation decisions which need to be made within a short executive summary.The result is that high-level stakeholders normally endorse whatever evaluation plan their staff suggest and, inappropriately, often assume that it will be outcome evaluation of some sort because they have no real idea of exactly what evaluation questions the evaluation will answer. Once evaluation planning has finished, the plan is then put into a desk drawer somewhere and a different set of documentation is used to control the evaluation implementation. This is because the original evaluation plan is too cumbersome for use for implementation and also because the evaluation starts to rapidly change as soon as it starts being implemented. As a consequence, the details in the evaluation plan quickly become out of date.
With the new visual evaluation plan approach, the evaluator creates an initial evaluation plan in DoView (they can use the Easy Outcomes template approach if they like, or come up with a modified structure that they like themselves). This visual evaluation plan is used by high-level decision-makers to decide on the priorities for the evaluation. I recently used this approach with a board using a visual evaluation plan to discuss evaluation priorities. In twenty minutes, using no paper (only the visual evaluation plan dataprojected in DoView) they were able to overview the evaluation and to have some strategic input into the evaluation direction. One of the board members commented on the ease of the process, saying: ‘where has this been all our lives?’ The visual evaluation plan once agree on is then used to implement the evaluation. It is modified every time any detail regarding the evaluation changes. It can be used to keep funders up-to-date as to where the evaluation is at at any point in time (they can be sent a PDF of the evaluation or directed at an intranet or internet address where the latest web page version of the model has been put up – both of these formats can be quickly created from within DoView).
Lastly, if staff change, then the visual evaluation plan can be used as a knowledge management tool and handed onto the new staff member to take up where the last staff member left off. Throughout this whole process, time is also saved because in normal evaluation planning and implementation, separate presentations need to be prepared for each meeting with stakeholders and decisions-makers based on the textual narrative planning and other documents being used. Because DoView can be used directly to present to decision-makers and stakeholders, this also saves a considerable amount of time.So, the concept of visual evaluation plans is now feasible and, if used, should save evaluators a great deal of time.
Paul Duignan, PhD (OutcomesBlog.org)
Comments»
no comments yet - be the first?