One I got my teeth into recently in an Op-Ed I wrote is an estimate regarding the cost of one of the many after-shocks which struck Christchurch NZ in its recent sequence of earthquakes. In this case, an estimate was made of one set of aftershocks which did not seem to take into account the fact that many of the buildings damaged in the after-shock had already been damaged in the earlier earthquakes. As one commentator said, ‘there are only so many times you can write off a building’!

This estimate was shot down by many people when it arose but the problem with junk economic estimates is that they tend to keep popping up like zombies like this one did in at a later date, but without the original criticism which put it in perspective. Anyway, check out the Op-Ed at http://www.outcomescentral.org/files/duignan299quake.pdf.

Paul Duignan, PhD. (Follow me on my OutcomesBlog.org; Twitter; or via my E-Newsletter; many resources at OutcomesCentral.org

Paul Duignan, PhD. (Follow me on my OutcomesBlog.org; Twitter; or via my E-Newsletter; many resources at OutcomesCentral.org

Meantime I have been implementing a commitment to keeping my work-life balance under control and spending time with the kids. So blogging has taken a lower priority for a little while. However, I’m planning on getting back to it to get out the interesting material and resources we have been developing. So watch this space.

Paul Duignan, PhD. (Follow me on my OutcomesBlog.org; Twitter; or via my E-Newsletter; many resources at OutcomesCentral.org

At the moment all that most people have to offer for ‘looking behind the numbers’ is the suggestion we verbally talk about what is behind the numbers and write text-based documents about what’s behind them. These can help improve ‘the numbers’ (i.e. the particular performance indicators being used in a particular case). However, what about instances where there are permanent problems with the numbers? i.e. important outcomes are not currently measurable with our current measurement technology and (equally importantly) how much we have to spend on measuring indicators? The numbers have a way of constantly reasserting themselves (reflected in the saying ‘what gets measured is what gets done’). This is because they are the only things on the page (on the PowerPoint, listed within the table in a printed document or in a traditional performance ‘dashboard’) when people sit down to make the important decisions.

My focus is on forcing a different discipline right into the heart of all discussions about performance. A discipline which provides a permanent reminder at the very moment when ‘the numbers’ are being considered that there is something ‘behind the numbers’. In my view, making sure that this discipline is followed at the moment when the numbers are being considered is critical.

It needs to be a robust discipline rather than just a lot of talk and background papers which all burn off when you are in a room with busy decision-makers and they have five minutes to make a decision based on whatever you give them. In my experience, general discussion and background papers about what is behind the numbers are largely irrelevant at such moments and just the numbers on their own come to dominate.

My suggestion is that all performance indicator systems should always consist of the numbers (the ‘indicators’) visually mapped back onto the underlying outcomes model of what is being attempted. It should be required that the visual model (including the measurable and the currently unmeasurable) is what is always placed in front of decision-makers at the moment they make strategic decisions. Using this discipline has the advantage that it both does justice to the numbers themselves (this is essential), but it also forces a transparent consideration of ‘what’s behind the numbers’ at exactly the same time.

This approach may initially be unsettling for some – those who like tidy lists of numbers with absolutely no hint that there may be important, but currently unmeasurable, outcomes which are being sought. But generally I find that those who start working with this new visual approach are relieved that finally there is some robust and disciplined way for them to bring the issue of the currently unmeasurable, but important, back into the strategic conversation.

Here is a paper on how this approach can be done easily:

Duignan, P. (2009). Mapping indicators onto a visual outcomes model. Outcomes Theory Knowledge Base Article No. 235. (http://knol.google.com/k/paul-duignan-phd/indicators-why-they-should-be-mapped/2m7zd68aaz774/72).

Here is a short YouTube video on how to do it:

Duignan, p. (2009). Painless Performance Indicators: A Visual Approach (YouTube Video) http://www.outcomescentral.org/performanceindicators5.html

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

I’ve been involved in a number of seemingly endless discussions and presentations about the pros and cons of randomized experiments and the rise of what are being called the Randomistas – those advocating for a much wider use of randomized experiments. In this post I want to talk about how we can move beyond these seemingly endless discussions.

In my experience, the outcome of such discussions inevitably are the following points made by the non-Randomistas: 1) randomized controlled trials have their place in certain situations; 2) they are not suited for determining causality in many other cases; 3) it is irritating that at the current time there are a number of people out there who are pushing randomized controlled trials as the ‘gold standard’ for impact evaluation with either an ignorance of their inevitable limitations or a willful unwillingness to be realistic about their limitations; 4) in the hands of people who do not understand anything about the technical issues involved the simplistic demand for more randomized controlled experiments is likely to lead to misallocation of evaluation resources to overambitious randomized experiments which may fail in the course of the experiment (and hence waste resources); lead to misleading results; or eventuate in a systematic bias in program delivery toward those programs which are easier to subject to randomized experiments.

Randomistas on the other hand will argue: 1) they acknowledge that there are circumstances where randomized experiments will not be appropriate; 2) however, randomized experiments are incredibly powerful (when they can be done properly) at determining causality and their being able to establish this is incredibly important in a number of cases to cut through the current confusion and political agendas which surround decision-making about resource allocation; 3) we should be pushing everyone to think very hard about when randomized experiments can be done so that we do not miss any opportunities to do so.

At the end of such discussions I generally find myself unfortunately in  general agreement with all of the above points!

In the light of ending up at the end of discussions like this feeling that way, the recent focus of my work has been on providing a practical way of moving forward in the face of the inevitability of discussions like the above. Ones from which no one ‘side’ will emerge the clear ”winner’. The end result of which is a lot of people spending a lot of time talking past each other.

The methodology I think we should use to move beyond endless discussions like this can be termed Impact Evaluation Design Assessment and it is a component of my Easy Outcomes approach.

This approach starts from first acknowledging the Randomistas point that it is worth at least a little thought in regard to any program to consider whether a randomized controlled experiment could be done on it. However such an assessment should be put in the context of also considering the other alternative impact evaluation designs to randomized experiments.

This leads to a relatively straightforward recommendation that any program assess a list of possible impact evaluation designs. I have a list of seven of them that I use which, importantly, includes a full range of impact assessment methods. The appropriateness, feasibility and affordability of each one of these design types should be documented as part of any monitoring or evaluation planning for any program. It seems to be that this gives us a practical tool which does justice to both sides of the debate while moving us forward beyond just a permanent state of discussing this issue.

The article which shows how to do it is Duignan, P. (2009).  Impact evaluation – when it should, and should not, be used. Outcomes Theory Knowledge Base Article No. 242. ( http://knol.google.com/k/paul-duignan-phd/impact-evaluation-when-it-should-and/2m7zd68aaz774/86 ). The list of seven impact evaluation design types is Duignan, P. (2005-2009). Impact/outcome evaluation design types. Outcomes Theory Knowledge Base article No. 209. ( http://knol.google.com/k/paul-duignan-phd/seven-possible-outcomeimpact-evaluation/2m7zd68aaz774/10 ).

An example of one page of the analysis within a visual evaluation plan is here http://www.outcomesmodels.org/models/communitycentral31-slices/communitycentral31-10.7.html .

Instructions on how to build visual evaluation plans like this are at http://knol.google.com/k/paul-duignan/-/2m7zd68aaz774/134

To do this type of analysis done for all programs would require some capability development around knowledge of impact evaluation but I don’t see that it is unrealistic to try to move to a point where it is done in regard to every program in some form. Or it might be done more generally for certain types of similar programs and individual programs could just point to the analysis.

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

It’s interesting the way that DoView is increasingly being used – in particular for strategic planning. A new DoView How-To article on Doing Strategic Planning Better: Using a DoView Visual Strategic Plan has just been put up on the DoView site to help people who are wanting to use the software in strategic planning.

A typical project I was involved in last week using DoView in this way was a straight forward strategic planning workshop. The traditional approach is to scrawl all over a whiteboard and flip charts and end up with a whole lot of material to process in some way into a written strategic plan. In contrast, what we did last week was to work directly with a visual outcomes model. One which had actually been prepared for the organization earlier as part of an exercise to work out key evaluation questions. This is the second time I’ve been involved in a situation where an Executive Team could immediately leverage off work down for evaluation planning and use it for their strategic planning. This is where my theoretical work has been heading and its great to see that it is starting to be translated into practic.  (The DoView Systematic Outcomes Management / Easy Outcomes process is a fully integrated process running from strategic planning, through monitoring and evaluation, right down to accountability specification and result-based outcomes-focused contracting – all done against the same visual model which encouraged organizations to leverage off work done for different purposes in this way).

In the case last week, that of a fairly small organization, the Executive Team just went through the visual model one page at a time and marked up the boxes within the model which show the steps that they regard as highest priority. Once that was done it set the strategic direction for the organization. More detailed planning is still to be done by mapping ‘projects’ (call them activities if you wish) onto the steps in the visual outcomes model. Once this mapping has been done, it is then obvious whether or not the organizational activity is aligned to its strategic priorities as visualized on the marked-up visual outcomes model.

What this all goes to show is the efficiencies which can be gained by using a standard DoView visual model underneath a whole range of organizational functions.

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

In the meantime, on the DoView site some new How-To Guides are starting to be put up. The first is on What’s a DoView Outcomes (Results) Model and Why Should I Use One? This is in response to requests from DoView enthusiasts who want to be able to refer people to a quick article about what an outcomes model is and why people should use one for all of their project and organizational planning.

What I’ve been progressively realizing with my involvement in the development of DoView outcomes software and the area of outcomes theory is that two things are happening at once. The first is that there’s a new piece of software around which can do some things that people have done in the past but they can now do more efficiently. The major one that DoView does is to help program evaluators draw better logic models in real-time in front of the groups they work with.

But what’s also happening is that DoView is being picked up my many other people, who are not using it for program evaluation at all. In combination with the work which is being done on outcomes theory, DoView is opening up an entirely new way of doing strategic planning within organizations. The most exciting thing is that the new visual approach to strategic planning which DoView is making possible, also allows a whole raft of other types of organizational activity (monitoring, evaluation, outcomes-focused contracting) to be done around the same visual outcomes models being used in DoView visual strategic planning. People are finding that this is an incredibly efficient way to operate.

While all this is very exciting stuff and we’re now involved in a number of really interesting projects pushing this work forward, the challenge of communicating what this new DoView / Outcomes Modeling movement is all about is now more complex. This is because it involves not just talking about how DoView can be used to do something which people are familiar with and is already being done (i.e. program evaluators drawing logic models). It also now involves explaining to a wide variety of people, with a wide variety of backgrounds, an entirely new (visual) way of doing things. Of course, this new way of doing things is all set out in detail on the outcomes theory site (e.g. in the article Results-Based Management Using the Systematic Outcomes Management / Easy Outcomes Process). But the problem is that busy practitioners (e.g. strategic planners for example), really don’t have enough time to get on top of all of this more theoretical material.

So what’s happening now is that a lot of this material is being simplified and integrated more with the use of DoView and put up on the DoView website. This is for those people who see what they’re doing as essentially using DoView for various purposes – rather than doing something like Results-Based Management Using the Systematic Outcomes Management / Easy Outcomes approach (a mouthful at the best of times!).

So the first How-To Guide on the DoView site referred to above, is the first of a number of articles on that site which attempt to explain this new movement promoting the use not only of DoView visual strategic plans, but the exciting integration (via visual outcomes models) of strategic planning with monitoring, evaluation, evidence-based practice and outcomes-focused contracting.

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

My problem with the notion that evaluation is always about the evaluator valuing the thing which is being evaluated is that I think in some cases it overstates the role that the evaluator is being asked to play. So while there may be some (many) situations in which the evaluator is asked to ‘value’ the program in terms of working out which are ‘good’ (e.g. valuable) and which are ‘bad’ (less valuable) outcomes, there are many other cases where the evaluator is just asked to work out if some outcomes have occurred and if they can be attributed to a particular program. But it is not really part of their brief for them to spend a lot of time working out what they think about the outcome in terms of ‘value’. An example would be in a welfare program for instance, where an evaluator may be asked if a program got single mothers with children over five back into work and off the benefit. No one may be particularly interested in their views as to whether this is a ‘valuable’ outcome or not, but I don’t think that peoples’ lack of interest in their views necessarily makes what they are doing not evaluation.

Of course, the evaluator can offer various opinions under an ‘unintended consequences’ heading if they think that the outcome is not a good idea (for instance, they may think that children with only one parent seeing even less of that parent is not a good idea), or they could refuse to take the job because they have some views on the matter. But I still think that the person who is doing the job of working out whether or not the outcome has been changed by the program is doing evaluation even though they are not doing much in the way of thinking about the extent to which that outcome is ‘valuable’.

I think that the stream of thinking that insists that evaluation always has to be about the evaluator establishing ‘value’ within an evaluation comes (or at least is consistent with the flavor of Michael Scriven’s (one of the formost evaluation thinkers) ideas about ‘goal-free’ evaluation. And, of course Michael Scriven often talks about the importance of valuing in evaluation. Goal-free evaluation makes the useful point that in some cases, if evaluators just concentrate on the objectives or goals specified by a program itself, they may be missing the wider picture of what it is that the program should be trying to do, not what it has decided is is going to try to do. Of course, if you take this position of what we could call ‘suspicion’ about the goals being struck by a program itself (and I am not saying that being suspicious like this is necessarily a bad thing in itself) there needs to be someone out there who can discern the ‘real’ or ‘genuine’ goals the program should be attempting to achieve. And of course this role seems to fall to the evaluator.

However, I just think that in many cases this is overinflating the role of the evaluator who is often just a functionary in a much larger system in which value has already been determined all over the place and it is not the evaluators role (or even often the role of those employing them) to establish what is and what is not true value in regard to program outcomes. Regardless of whether we think such a thing is desirable or not, I think that we need to acknowledge that in many situations people will be doing things they call evaluation and will call themselves evaluators but they will not be doing a lot of specific work on determining ‘value’.

And so as a consequence, that leads me to not want to demand that a definition of evaluation must involve the notion of establishing value. It definitely can be something which happens within evaluation, but at the moment I can’t see that it is something that has to happen for an activity to be called evaluation. There you are, I’ve finally got it off my chest! Please if you have a moment and think otherwise, have a go at my argument because I may well have it wrong.

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

In my work I currently believe that there are just models of the way the world works and that we can drill these down to lower and lower levels and that we can map onto them things like indicators and evaluation questions and, importantly for this discussion, the particular focus of different projects. See examples of the types of models I’m involved in drawing at OutcomesModels.org.

However, the Harvard Family Research Project makes a distinction as follows: “A theory of change identifies the process(es) through which a given type of social change is expected to occur. A theory of action maps out a specific pathway in that theory of change, or an organization’s role with respect to achieving that change, based on an assessment of how it can add the most value to the change process.” Ref

So there’s no problem defining different levels of analysis I suppose, for instance that is what is done in the distinction between a strategic plan (high-level) and a business plan (lower-level) within strategic planning. However, what I am interested in is whether there is any actual conceptual difference in the way such models are drawn.

In my practice at the moment anyway, I just simply approach all of the different types of outcomes models which people talk about (logic models, theories of change, theories of action, intervention logics, strategy maps etc) as causal models of the way we would like things to have turned out in the world by the time we finish our intervention. My approach is to only make distinctions if we really need to, otherwise our field of work gets cluttered with all sorts of distinctions (for instance see simplifying drawing logic models with stakeholders as an example of this parsimonious approach). I’m however happy to introduce distinctions if it is essential to make such distinctions.

So in my approach we can define a set of conventions for modeling these things, and develop standards for drawing them and resources for helping people model them in the way they should be modeled as I discussed in my last blog posting on the evolution of  logic modeling within the evaluation discipline.

Thinking more about the Theory of Change versus Theory of Action distinction and whether it is essential to make it, the Harvard people say, a Theory of Action maps out ‘a specific pathway in that theory of change, or an organization’s role’.

Probably the way they are doing this in practice is exactly the way I would do it, I haven’t had a chance to look in detail. However, just one point is that in the way in which I work, I simply continue to developed the outcomes model down to lower and lower levels using exactly the same approach – i.e. building it as a general ‘world centric’ (as opposed to ‘program centric’) outcomes models (meaning that it is allowed to contain steps and outcomes which are the focus of more than one project or organization). I then just map onto this the particular focus of particular projects. See this short video here for how you do such mapping and if you have more time look at the end of the two part video in this article on a maternal health project on how the same approach can be used to identify gaps and overlaps when there are a number of projects focusing on similar outcomes.

This means that you don’t get a proliferation of many different ‘Theories of Action’ if you want to call them that – i.e. one for each project which are drawn at too high a level. What do I mean by ‘too high a level’? What I am getting at here gets a little technical but hang in with me  -  its about the issue of attribution.

At a certain level most (if not all) of the steps which appear in an outcomes model will be attributable to a particular project. By attributable I mean that it will be self-evident that they have been caused by that project (if you want more on attribution look at an article on the features of steps and outcomes in outcomes models).

But as we move up the outcomes model we will usually find that more and more of the steps within the model are not attributable to just one project or factor (e.g. other projects influence them or they are also influenced by external factors). Now I believe that when we are modeling (say for a particular project) it is important that we continue to model all of the steps and outcomes for a particular level within the model  – not just the ones which are attributable to a particular project. That being the case it seems to me that at such a level of modeling the Theory of Change and the Theory of Action are exactly the same thing – i.e. a model of the way we would like the world to have turned out if our intervention was successful. (i.e. we just use a common set of standards for drawing the model at any level – such as this 13 Tips one-pager here which summarizes the standards for people actually drawing models).  It is just the case that at very low levels within the modeling (if you take modeling to that level) that all of the steps in the model are likely to be ones which are just attributable to a particular project.

So for me currently there is no theoretical distinction which needs to be made between a Theory of Change and a Theory of Action. The only difference is the level to which they are modeled. I’d love to hear any comments on whether my approach here makes sense.

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).

The article argues that if we want to work out how evaluators should draw logic models, then we need to define the purposes for which we want to use logic models before we can do so. It discusses the wide range of ways in which logic models can be used and refers to a set of conventions for drawing logic models which have been developed to develop logic models (alternatively termed outcomes models) which are fit-for-purpose for this variety of uses. From these conventions a set of practical standards has been developed, one page tip sheets incorporating these standards, and other resources for drawing such models. This article is, in part, based on a presentation I did at the recent American Evaluation Association Conference (Rejecting the traditional outputs, intermediate and final outcomes logic modeling approach and building more stakeholder-friendly visual outcomes models. American Evaluation Association Conference 2009, Orlando, Florida, 11-14 November 2009. Abstract http://www.eval.org/search09/session.asp?sessionid=7424&presenterid=630).

Check out the article and if you have any comments, put them at the bottom of it:

Duignan, P. (2009). The evolution of logic models (theories of change) as used within evaluation. (http://knol.google.com/k/paul-duignan-phd/the-evaluation-of-the-use-of-logic/2m7zd68aaz774/135).

Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).