Outcomes police, do they exist? October 7, 2007
Posted by Paul Duignan in : Outcomes theory, Standards , trackback
Several days ago I came across an article by John Day asking the question - Accounting Police, do they exist? In it he talked about the role of the Accounting Standards Board (FASB) in the U.S. Members of this board, ‘Go through a lengthy process of analyzing and reviewing problems in the accounting field that are brought to them. After much thought, they will make a pronouncement as to what they think the new or revised way of approaching the treatment of an accounting issue should be.’
There’s no such body for the outcomes field, I think that there should be one. The absence of people doing this hard (it’s not necessarily the most exciting work in the universe!) but necessary work in the outcomes area means that we don’t have standards and conventions - people simply make it up as they go along - some times they get it right, sometimes they don’t. It would make life a lot simpler for everyone who has to work with outcomes systems if we had a set of well thought through rules for building and using them.
Backing up accounting systems there’s accounting theory, trained and certified accountants, accounting standards boards and accounting conventions. Conventions are designed to help people deal with the difficult situations which arise in working with real world systems and which are much better dealt with by collectively agreeing on an approach rather than everyone just muddling through and often doing things in quite different ways. Behind the outcomes area we have almost none of this. What often happens is that someone who’s doing some other type of work (e.g. policy or operations) is asked to ‘measure outcomes’, ‘report on outcomes’ or to set up a performance management system. They do their best, but often they simply don’t understand the basic principles of outcomes systems which need to be followed if such systems are going to work.
The popular rhetoric about the importance of outcomes and results makes the situation more difficult in some ways. It seems simple common sense that you should measure the outcomes of whatever you’re doing and see if you’re improving them. However, its more complex than that. In situations where we can take that approach, we simply go ahead and do it - no problem. But in most situations these days where people are being asked to come up with outcomes, the fact that they have not been providing information on outcomes all along, reflects the fact that there are difficulties in doing so in their particular area of work. For instance, it may be hard to measure outcomes, or if you can measure them, it may be hard to attribute changes in them to particular players. These sorts of difficulties lie behind the fact that outcomes have not been routinely provided in the past. Administrators over the years have not been idiots, they’ve always been able to see the benefit of having outcomes data where they could get it. If it was easy to collect outcomes in these areas then long ago, funders and those higher up the administrative chain would have been likely to ask for outcomes data to be reported on.
Of course, it’s good to start measuring and reporting on outcomes in these areas but often what happens is a naive approach to such reporting. For instance, often outcomes data is collected and reported but there’s no way of knowing whether particular programs or other interventions actually caused any reported changes in outcomes. Sometimes players are rewarded or punished for changes in outcomes data, but those doing the punishing or rewarding remain forever ignorant about exactly what it was that actually did, or did not, change the outcome (in technical terms these are called not-necessarily attributable indicators - they’re great to measure for strategic purposes but only under certain contracting arrangements is it appropriate to hold people to account for these, which in a contracting setting are also called not-fully controllable outcomes).
Outcomes theory is my attempt to start specifying the theoretical basis of all outcomes systems and Systematic Outcomes Analysis and its user-friendly companion Easy Outcomes attempt to provide ways of applying the insights of outcomes theory. I’ll be doing a substantial update to the outcomes theory website in about a month or so as the current site is from 2005 and doesn’t include my latest work in this area.
So, in conclusion, I’d like to see ‘outcomes police’ of some sort out there sorting out the wild west of outcomes systems which I have to deal with daily. Exactly where such work would take place and under what institution it would be located present significant impediments to us seeing a solution to this one in the immediate future.
Paul Duignan (outcomesblog.org)
Comments»
no comments yet - be the first?