Castles of sand – cost benefit modeling September 30, 2007
Posted by Paul Duignan in : Easy Outcomes, Economic analysis, Outcomes models, Systematic Outcomes Analysis , trackback
A while ago I came across a report of a cost-benefit analysis on climate change in which the author of the analysis admitted that his model may come in for some heavy criticism because it didn’t include any cost for sea level rises. How cruel and heartless of his critics. I don’t know who paid for this particular report or why it was done, so I don’t want to comment on it at all. However cost benefit analyzes which leave out or minimize important costs are a well used weapon in the lobbyist’s armory. Policy makers, the media and the public only have time to catch the bottom line – the cost will be this or that much and then move on.
Cost-benefit analyses which leave out major costs are, well, simply no longer full cost-benefit analyses. They’re better described as ”analyses of some costs and some benefits’. Or perhaps, in some cases where they’ve been paid for by someone with an economic interest in a certain result from the analysis, ‘analyses of costs and benefit omitting one or more costs with the result that the final conclusion will be biased in the direction desired by those who purchased the report’.
What does outcomes theory have to contribute to tidying up cost-benefit studies which are actually just special pleading dressed up as sound economic analysis? Firstly, it argues that best practice would be for any cost-benefit report to show the reader the underlying outcomes model which is being modeled in the report. This needs to be presented in a form (visualized for instance) where the reader can quickly grasp the underlying outcomes model and quickly see which outcomes (costs and benefits) are and are not included in the analysis. If this is not done clearly in the report, the reader has to construct in their own mind an hypothesized outcomes model of what they think the topic looks like. They then need to assess whether the analysis has included all of the relevant major outcomes.
Ideally in the future, outcomes theory sees us working towards standardized underlying outcomes models which would be used by a number of analyses throughout a sector and which would represent a consensus on what are the important underlying outcomes are in regard to a particular topic. Such an approach would provide even more assurance that an analysis was not just cherry picking which outcomes to include and ending up biasing the results in a certain direction. In drawing the outcomes model for a topic it’s important that it’s drawn about the real world not just about what’s measurable or attributable to particular players otherwise it will not give us a clear sense of what has been left out of the cost-benefit analysis (the Easy Outcomes outcomes model guidelines based on the Systematic Outcomes Analysis outcomes model standards can help in this regard).
The second contribution of outcomes theory to thinking about cost-benefit analysis is to help ensure that we’re clear about the evidential robustness of the estimates made in an analysis. There’s an old joke about an economist being someone who’ll offer to estimate your telephone number for you. Economists are presented with a major challenge in doing cost-benefit analyses. The problem is that they need estimates related to all major outcomes to plug into their analyses, otherwise people like me will criticize them for leaving some important ones out, but at the same time the evidential basis for making estimates varies widely for different outcomes in an outcomes model. Hence you can understand the propensity of some to offer to estimate your telephone number.
Outcomes theory would argue that what needs to happen before a cost-benefit analysis is done is to survey the underlying evidential basis for the estimates that will need to be made. The analogy here is with building a house. The cost-benefit analysis is the house which is going to be built on on top of a particular terrain. Before the house is built an essential piece of work needs to be done – having an engineer survey the underlying terrain. This engineering analysis will show where the house is able to be built on sound foundations and where it will be floating on thin air. It will be floating on thin are in those areas where there’s no sound evidential basis for the estimates which will have to be made to do the cost-benefit analysis.
Many cost-benefit analyses are bizarre constructions with some estimates which are soundly based on evidence and other estimates which have little or no evidential basis at all. How can we survey the evidential landscape? Systematic Outcomes Analysis provides (and its user friendly version Easy Outcomes) provides one way of doing this. At the end of doing such an analysis there should be a clear picture of what is known, what is not know and what its going to be feasible and affordable to know in the foreseeable future.
This can let us be very clear about exactly the type of economic analysis we should be doing on top of this evidential foundation. Systematic Outcomes Analysis divides possible economic analyses up on the basis of the type of evidence that is available for making estimates of effect sizes for particular interventions. For instance it proposes that where there is an insufficient evidential basis for making an effect size estimate that a series of hypothetical effect sizes could be modeled and still provide assistance to policy makers without having to make the unrealistic claim that an actual effect size can be established. Of course this is complex to do in practice in cost-benefit analyses where you are dealing with a number of different estimates.
This is partially picked up in good cost-benefit analyses where what is called a sensitivity analysis is done looking at what effect varying particular estimates has on the final result. Combining this with the Systematic Outcomes Analysis approach means that the place you would want to look very carefully at in a cost-benefit analysis is where there is an estimate which when varied will have a large effect on the final result. If this is an estimate which Systematic Outcomes Analysis shows does not have a sound evidential basis then the use of the cost-benefit analysis for decision making must be questioned.
Overall outcomes theory points to the importance of clarifying what you know, what you don’t know and what you can possibly know so you are clear about the evidential basis on which you are trying to make decisions. It also suggests that it is better to be clear that you are making a decision under uncertainty rather than being fooled by a cost-benefit analysis which claims to reduce your uncertainty when, in fact, from an evidential point of view it is merely a castle build of sand.
Paul Duignan (outcomesblog.org)
Comments»
[...] you need such a model; to do a cost effectiveness/cost benefit study you need such a model (see my posting on how outcomes models should underly all cost benefit modeling); to sensibly report on indicators [...]