Clarifying program outcomes to reduce ‘talking past each other’ about program effectiveness September 27, 2007
Posted by Paul Duignan in : Outcomes theory & the news, Outcomes models, DoView , trackback
This morning when casually looking through old Washington Post articles which make reference to the General Accountability Office (GAO). (Hey: people have all sorts of hobbies and no one challenges them about it - beer tab collecting etc.). I came across an article about DARE - the Drug Abuse Resistance Education program. This is a program which puts law enforcement personnel into schools to talk to school children about drugs. The wisdom of funding DARE and other drug education in schools is the source of endless debate in evaluation and public health circles as important summaries of the evidence (such as the 2003 GAO report) referred to in the article fail to find evidence of the program having an effect on illicit drug use.
One outcomes theory take on this issue is what I would could call a ‘potential intervention potency’ one - you don’t hand out aspirin for cancer. School programs are politically expedient, attractive to fund, relatively easily implemented, high ‘front page photo opportunity’ and high public ‘feel-good factor’ ways to try to address social problems of all kinds; problem which will probably take much more than a few hours with kids in the classroom to solve. So there’s going to be an inbuilt bias for them to be offered up as the initial response to a wide range of social problems. That been said, if such programs about particular topics are shown to work (and the GAO report says that there are programs that do show evidence of working) and they are, importantly, shown to be an economically efficient approach to a problem, then let’s roll them out. Also, as discussed below, there may be other reasons for running such programs even if they don’t impact directly on illicit drug usage.
However, I don’t want to get into the middle of the debate over such programs. What I want to talk about is a technical issue around program outcome specification. The pattern of argument in the Washington Post article is representative of how arguments about negative findings about program outcomes often go. After talking about the negative evaluation findings an advocate of the program is quoted as saying: ‘. . . the main benefit of the program is that it allows children to have contact with law enforcement officials in a positive, educational environment’. That is followed with a different commentator making a counter attack on DARE: ‘It may be that we want it [DARE] to work so badly that we give it the benefit of the doubt. . . but we should be using programs that have proven they’re effective or that we have good reason to believe would be effective, which does not include DARE.’
Now what’s happening here at this point is that the two people quoted are talking past each other. They are talking about two entirely different outcomes for DARE. The DARE defender is talking about contact with law enforcement officers in a positive environment while the GAO report, and presumably the second person quoted in the article, is talking about reductions in illicit drug use.
Looking at this debate as it’s represented here, nothing’s really going to happen and it will just track on year after year. It’s tiring - I’ve actually been looking in on this sort of debate about evaluation of drug education programs in schools for over a decade. From a technical point of view this argument is a waste of time and likely to go nowhere because it’s not been set up in terms of a clear specification of the DARE outcomes that are being argued about. The abstract of the GAO report does not mention anything about law enforcement officers relating to students in schools as an outcome. The first commentator quoted in the article sees this as the ‘main benefit of the program’.
So where do we go from here? From an outcomes theory point of view we’re simply interested in setting up the debate clearly so that it has any possibility of moving forward in a coherent way. Presenting the outcomes graphically in DoView as above would be a start if you were wanting to have a sensible discussion about this (I know it’s unlikely that this would work in the format of a Washington Post article) but if you were having the debate in another more formal forum.
Having made it clear that there are at least two different outcomes for DARE under debate where would the argument go from here? If they wanted to, DARE supporters could say either that improving law enforcement officers’ relationships with young people is the main focus of DARE (as seems to be being said here) or that it is an additional focus of DARE which somehow compensates for the apparent lack of evidence of an effect on illicit drug use. Of course, this would open up a discussion of what is the most effective and economically efficient way of improving the relationship between law enforcement officers and students. This would be an interesting discussion ultimately amenable to further evaluation research as to whether programs in schools about drugs is the best way to do it. DARE would presumably argue that having the focus on drugs is a good topic which will engage students in a way that other topics would not.
The opponents of DARE could argue that the secondary objective of DARE (improving relationships between law enforcement officers and young people) is just being pulled out of the hat whenever DARE is up against the wall and that DARE is just using it to avoid facing the fact that its actual outcome (reducing illicit drug usage) is not being achieved. Some of them may also say it’s fine if that’s seen as the outcome of the program, however it shouldn’t be receiving any drug problem prevention funding in those instances where it is receiving any such funding.
Anyway, from an outcomes theory point of view this would be a better basis for the debate than the current ‘talking past each other’ we see in articles like the one discussed here. Formally structuring the debate using a simple visual representation of outcomes may help to move it on. It’s important that we make progress on this, a lot of good people put tremendous work into programs like DARE and a lot of people contribute to their funding because they want to improve the way our societies handle the drug issue. If we don’t move the discussion on we’ll simply have to continue listening to people ‘talking past each other’ on this topic in the Washington Post and other media for years to come.
Paul Duignan (outcomesblog.org)
Comments»
Speeding: An Excellent Law Destroyed By Dumb Enforcement…
The offence of exceeding the speed limit was sensibly created to improve road safety and offer guidance to drivers. When it is applied with intelligence it can do exactly that, and indeed generally until about 1985 that’s exactly what we had….