Castles of sand – cost benefit modeling September 30, 2007
Posted by Paul Duignan in : Easy Outcomes, Economic analysis, Outcomes models, Systematic Outcomes Analysis , 1 comment so far
A while ago I came across a report of a cost-benefit analysis on climate change in which the author of the analysis admitted that his model may come in for some heavy criticism because it didn’t include any cost for sea level rises. How cruel and heartless of his critics. I don’t know who paid for this particular report or why it was done, so I don’t want to comment on it at all. However cost benefit analyzes which leave out or minimize important costs are a well used weapon in the lobbyist’s armory. Policy makers, the media and the public only have time to catch the bottom line – the cost will be this or that much and then move on. (more…)
Clarifying program outcomes to reduce ‘talking past each other’ about program effectiveness September 27, 2007
Posted by Paul Duignan in : DoView, Outcomes models, Outcomes theory & the news , 1 comment so far
This morning when casually looking through old Washington Post articles which make reference to the General Accountability Office (GAO). (Hey: people have all sorts of hobbies and no one challenges them about it – beer tab collecting etc.). I came across an article about DARE – the Drug Abuse Resistance Education program. This is a program which puts law enforcement personnel into schools to talk to school children about drugs. The wisdom of funding DARE and other drug education in schools is the source of endless debate in evaluation and public health circles as important summaries of the evidence (such as the 2003 GAO report) referred to in the article fail to find evidence of the program having an effect on illicit drug use. (more…)
Avoid being an outcomes model ‘Go-Between’ September 26, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Outcomes models, Standards, Using the approach , add a comment
A while ago a colleague recounted to me how they’d ended up pulling out their hair because they found themselves in a ‘Go-Between’ role when drawing an outcomes model (also called program logics, results chains, strategy maps, ends-means diagrams). You need to try to avoid this at all costs, although when dealing with high level stakeholders it’s often not easy to do so. I found myself in this role on a major project a while ago and I certainly didn’t enjoy it.
(more…)
Slicing up the world September 25, 2007
Posted by Paul Duignan in : DoView, Outcomes models , 2comments
The other day I was talking to Laurie Porima, an evaluation colleague, about building outcomes models with DoView outcomes software. We started discussing the best way of dividing an outcomes model (program logic, results chain, strategy map, ends-means diagram) up into sections. Obviously the real world of causality is not divided up into tidy conceptual pieces suitable for humans to quickly grasp. Many traditional approaches to outcomes models do not have to grapple with this problem because they limit themselves to a single printed page. There’s no question of having to divide the model up in any particular way. However if you allow your models to be larger than a single page – which seems to me to be essential if we are to develop models which can even start to do justice to the causal complexity of the world we’re trying to operate in – you need to think about how you can visualize this. (more…)
Typical media story about outcomes – crime stats increase September 24, 2007
Posted by Paul Duignan in : Outcomes theory & the news , add a commentThe issues dealt with by outcomes theory – which looks at the ways we think, discuss and argue about outcomes – are central to many of the political debates which we read about daily in the media. For instance today (24 September 2007) the Washington Post carries a story with the headline FBI Report: Violent Crime on the Rise. It’s fairly typical of the type of story we see. From an outcomes perspective what are the important elements of these types of stories? Firstly, there’s a claim by the media that there’s some ‘news’ in what they’re reporting – that something is changing. The media need this, basically what they sell is news of change – if nothing changes there’s nothing to tell us about, there’s no ‘news’. In this case the subtitle is: ‘First significant increase in homicides and robberies since 1993 continues.’ Outcome measures can go up and down due to statistical variation, is this change actually significant from a statistical point of view? There’s no way to tell from the story, I presume they’re not referring to statistical significance in the title. If they were they would need to make this clear in the article if we are to be able to have any basis for deciding whether we should regard this as an actual change or not. (more…)
When should measurements appear in an outcomes model? September 21, 2007
Posted by Paul Duignan in : Measurement , add a commentOne of the important principles of outcomes theory reflected in the Easy Outcomes approach is that measurements should be kept separate from outcomes in outcomes models. DoView outcomes software is structured to let you do this. This is important because it frees you up from just focusing on the measurable and encourages you to focus on the important – the outcomes you’re trying to achieve. Measurement is always vital, but it should follow on after you’ve specified the outcomes you are trying to achieve. The classic example of where a measurement has inappropriately taken over an outcomes model is Gross Domestic Product (GDP). Sometimes targets are set for a country in terms of the GDP league table. A country would be better off identifying an economic, social and environmental outcome and then look at the best way of measuring these rather than single-mindedly pursuing growth in GDP at the expense of all else.
Opening up a gap between outcomes and their measurement allows the possibility that better means of measuring an outcome may be developed in the future. This also opens up the discussion of what is described in some of the social sciences as the reliability and validity of measures. Reliability is how reproducible a measure is – will different observers come up with roughly the same levels on a measure while validity is whether or not the measure is actually measuring the outcome. In the real world there is often a trade-off between these two and its often important to be aware of this measurement issue. (more…)
What’s the Easy Outcomes site all about? September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning , add a commentI’ve realized that I’ve not put up a post about the Easy Outcomes site even though it’s been up for a few weeks. Easy Outcomes takes the Systematic Outcomes Analysis approach and applies it using DoView outcomes software. We developed it in response to feedback that the Systematic Outcomes Analysis site is a bit intense for those who just want to work out how to develop a good evaluation plan, but don’t want to get buried in the detail of the approach too soon. (more…)
New Systematic Outcomes Analysis case study – evaluating an academic research rating scheme September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Evaluation planning, Systematic Outcomes Analysis , 1 comment so farI’ve recently put up the link to another case study on the Systematic Outcomes Analysis site. This is a report detailing an evaluation and monitoring plan for the Performance Based Research Fund (PBRF). The PBRF is a national academic research output assessment system and the monitoring and evaluation framework was produced for the government organization responsible for the tertiary sector – the Tertiary Education Commission.
The report is a good example of a comprehensive use of Systematic Outcomes Analysis (in the report the approach went by one of its earlier names – REMLogic). One of the interesting things about the PBRF is that it’s an evaluation system in its own right and developing an evaluation framework for an evaluation system presents an interesting technical challenge. I’ll put up a blog post about that challenge some time in the future. (more…)
Reorganizing outcomes sites into Outcomes Central September 18, 2007
Posted by Paul Duignan in : Blog info , add a commentWe’ve just reorganized our outcomes sites to make them more accessible to users. Outcomescentral.org now lists and describes our integrated set of six outcomes sites – a one stop shop setting out just about everything one might want to know about outcomes. Outcomes theory – the concepts behind all outcomes systems; Systematic Outcomes Analysis – a comprehensive applied system for working with outcomes; DoView - outcomes software for working with outcomes and evaluation plans; Easy Outcomes – a user-friendly approach applying Systematic Outcomes Analysis using DoView; Outcomes Models - a collection of outcomes models (program logics, strategic maps, ends-means diagrams) you can download, play with, amend and use for your own projects; this Outcomes Blog you’re reading here – new thinking about outcomes, strategy and evaluation; and finally a Frequently Asked Questions (FAQ) page which is designed to head you in the right direction to get what you want from our set of six integrated outcomes sites. (more…)
Outcomes: Keep it simple – but not simplistic! September 14, 2007
Posted by Paul Duignan in : Communicating outcomes models, Outcomes models, Standards, Strategic planning , 1 comment so farI was in on an interesting conversation a while ago between a strategic planner and a communications specialist in a larger organization. The issue they were discussing was how to communicate to staff the outcomes model which had been developed by the management team for strategic planning purposes. The communications person was saying that at first sight the outcomes model seemed too complex to communicate to staff and that they thought that it should be simplified down to four or so points which they could get through to staff. In fact, as far as outcomes models go, I didn’t think it was a very complex outcomes model at all. I don’t know what happened in the end in that organization - Comms may have worked out a clever and clear way of communicating all of the important elements in the full outcomes model to staff. However, the conversation led me to reflect on the issue of keeping things simple in outcomes modeling. (more…)