Tracking jobs created under the U.S. Recovery Act – when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Indicators, Measurement, Outcomes systems architecture, Outcomes theory & politics, Outcomes theory & the news, Reporting systems, Using the approach , trackbackThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be.
A recent media story questions how easy it’s going to be to get figures for the number of jobs created as a result of the U.S. Recovery Act. It includes the normal kinds of statements you get in situations like this, such as it is ‘essential that federal agencies and states be able to provide citizens with understandable, accurate reports about how Recovery Act funds are being spend’. And who could disagree with this? However, the problem is that I think that most people will read ‘accurate reports’ to mean that an overall measure of how the number of jobs has changed over the course of the program will be provided together with a claim that all or some of whatever increase there has been is attributable to the Recovery Act.
However, the media story casts doubt on the ease of actually doing this. There’s a comment from the Office of Management and Budget (OMB) saying: ‘we’re doing our best to not require recipients to be in the verification business’ – meaning that they do not want them to be involved in extensive checking regarding the accuracy of reports of created jobs. This sounds like a pragmatic approach, but if you take such an approach, you can’t at the same time somehow expect to be able to necessarily rely on the accuracy of the results which are returned. And, in fact, within the article there are comments about incentives for those in receipt of the grants to inflate the estimates of the number of jobs created which caste doubt on the likely accuracy of the estimates.
I don’t know anything about how accurate data collection will be in this particular instance and don’t want to assume that it will, or will not be, possible to develop a measure. However, give that there has been some questioning of the possible accuracy of the figures in regard to this program I want to use this media story to explore the scenario that for some programs it may not be possible to get accurate data – is this even considered as a possibility in peoples’ thinking about programs like this? Where it is the case (it may or may not be in regard to the Recovery Act funding) how should we deal with monitoring the performance of such programs?
In order to make rational decisions about the type of monitoring and evaluation that should be done for a program like the Recovery Act we would need to have considered how accurate the jobs data is going to be without any major verification effort. We need to at least consider the fact that it may be that the ‘unverified data’ is so bad that it is a waste of time collecting it (as I said I am not saying it is, or is not, in this particular case, just that we need to consider the possibility that it could be bad).
I don’t think that this is a possibility which many of those involved in this issue are open to considering in programs like this. I think that the default assumption is that it is essential that figures are provided and this assumption is so ingrained that the people who hold it are simply forced to not consider the possibility that the end result may be an unreliable set of data which cannot tell us what people are hoping it can – i.e in this case the number of jobs which have been produced by the program. The point I want to make is that even if we could not produce such a set of data, there are other things we could do to collect performance information about the program.
Of course, there are all sorts of political pressures which I presume make it impossible to simply not collect such data where it is likely to be unreliable. However, if I were in charge of the world I would re-set expectations in regard to what we can measure and attribute to programs. I would get people to start from the assumption that when looking at any program we need to carefully consider the possibilities for collecting each of the five possible types of evidence we can get about the performance of a program (the five building blocks of outcomes systems). We can’t assume before the fact that we’re definitely going to be able to provide any one of these types of evidence at any particular level of detail.
The issue I am highlight using the case of the Recovery Act, to put it in technical terms, is that it seems to have been assumed that the third building-block (a demonstrably attributable indicator) will be able to be provided at the highest level (i.e. jobs created) and it was this possibility which was questioned by the media article.
Taking the approach I am suggesting, the design of the system to measure the performance of the program would consist of an appropriate mix of information it is feasible and affordable to collect from each of the five building-blocks. So, in a case like this, if it was actually established that it would be difficult to collect high-level demonstrably attributable indicator information – i.e. jobs created (for instance because that it would require a massive verification effort to make sure that it was accurate); other building-blocks would have to be relied on. For instance good results/outcomes models (building-block one); overall tracking of not-necessarily demonstrably attributable job creation statistics (building-block two); collection of lower-level demonstrably attributable indicators (building-block three); and good formative evaluation to ensure the application of best practice to maximize the chances of success (building-block five).
Paul Duignan, PhD
Outcomes and Evaluation Blog (OutcomesBlog.org)
Comments»
no comments yet - be the first?