Is it the role of an evaluator to always ‘value’ what they are evaluating? January 21, 2010
Posted by Paul Duignan in : Evaluation debates, Use of terms , trackbackI’ve had occasion recently to need to think about whether or not the notion of ‘valuing’ something is always an essential part of evaluation. To question this may seem a heresy to some evaluators who see this as the defining aspect of evaluation (for instance as opposed to ‘research’ where they don’t see such valuing as needing to take place). I’m not definite in my thoughts on this issue and below just want to float the argument which has been rattling around in my head for a while and which I have not had a chance to get down in writing to see if it can be shot down – in which case I will change my mind.
My problem with the notion that evaluation is always about the evaluator valuing the thing which is being evaluated is that I think in some cases it overstates the role that the evaluator is being asked to play. So while there may be some (many) situations in which the evaluator is asked to ‘value’ the program in terms of working out which are ‘good’ (e.g. valuable) and which are ‘bad’ (less valuable) outcomes, there are many other cases where the evaluator is just asked to work out if some outcomes have occurred and if they can be attributed to a particular program. But it is not really part of their brief for them to spend a lot of time working out what they think about the outcome in terms of ‘value’. An example would be in a welfare program for instance, where an evaluator may be asked if a program got single mothers with children over five back into work and off the benefit. No one may be particularly interested in their views as to whether this is a ‘valuable’ outcome or not, but I don’t think that peoples’ lack of interest in their views necessarily makes what they are doing not evaluation.
Of course, the evaluator can offer various opinions under an ‘unintended consequences’ heading if they think that the outcome is not a good idea (for instance, they may think that children with only one parent seeing even less of that parent is not a good idea), or they could refuse to take the job because they have some views on the matter. But I still think that the person who is doing the job of working out whether or not the outcome has been changed by the program is doing evaluation even though they are not doing much in the way of thinking about the extent to which that outcome is ‘valuable’.
I think that the stream of thinking that insists that evaluation always has to be about the evaluator establishing ‘value’ within an evaluation comes (or at least is consistent with the flavor of Michael Scriven’s (one of the formost evaluation thinkers) ideas about ‘goal-free’ evaluation. And, of course Michael Scriven often talks about the importance of valuing in evaluation. Goal-free evaluation makes the useful point that in some cases, if evaluators just concentrate on the objectives or goals specified by a program itself, they may be missing the wider picture of what it is that the program should be trying to do, not what it has decided is is going to try to do. Of course, if you take this position of what we could call ‘suspicion’ about the goals being struck by a program itself (and I am not saying that being suspicious like this is necessarily a bad thing in itself) there needs to be someone out there who can discern the ‘real’ or ‘genuine’ goals the program should be attempting to achieve. And of course this role seems to fall to the evaluator.
However, I just think that in many cases this is overinflating the role of the evaluator who is often just a functionary in a much larger system in which value has already been determined all over the place and it is not the evaluators role (or even often the role of those employing them) to establish what is and what is not true value in regard to program outcomes. Regardless of whether we think such a thing is desirable or not, I think that we need to acknowledge that in many situations people will be doing things they call evaluation and will call themselves evaluators but they will not be doing a lot of specific work on determining ‘value’.
And so as a consequence, that leads me to not want to demand that a definition of evaluation must involve the notion of establishing value. It definitely can be something which happens within evaluation, but at the moment I can’t see that it is something that has to happen for an activity to be called evaluation. There you are, I’ve finally got it off my chest! Please if you have a moment and think otherwise, have a go at my argument because I may well have it wrong.
Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).
Comments»
Paul,
I see 2 issues here:
1) How do we define this construct “value”? For instance, does a formative evaluation, aimed at examining program implementation for the purpose of program improvement assess the *value* of the program? I would say that even though this evaluation does not examine higher level outcomes (program impact on the target population), it is still examining certain dimensions of program value: any program that is poorly managed and executed is not likely to provide much value to anyone. So, even a formative evaluation examines program value from my perspective, and can contribute to improving the value of the program, but I’m sure that not all would agree with that.
2) Is it essential that the evaluator personally determine the value? What if I were to say that it is *always* the evaluator’s job to generate information that will be useful in determining the value of a program, product, or policy, but it is not essential that the evaluator him/ herself makes this determination? Thus, even if the evaluator does not personally or directly establish the value, he or she must always identify and assess the issues that will need to be understood in order for *someone* to judge value.
Just my 2c, since you asked so nicely.
Mark
Thanks Mark,
I was thinking a little about both of these points that you raised and they are really interesting. We could go into what we mean by ‘value’ in a lot of detail and I would definitely agree with your first point that the type of ‘value’ you are talking about here would be a type of value that would be investigated in any formative evaluation, so I think that we could say that it would be essential to at least formative evaluation? But I would probably call this value with a little ‘v’ in the sense that it is pretty low level stuff and not highly contentious. So therefore maybe we should contrast this with the type of higher level ‘value’ I was trying to talk about which is all about outcomes which was what I was focusing on in the post.
Regarding your second point – that is a good angle to take and in fact at one stage in drafting the post I was trying to argue that in empowerment or stakeholder evaluation the evaluator may not strike the value, but I then deleted it because of the argument you are mounting which I think is a strong one that, regardless of whether the evaluator strikes the value, they, in some evaluations, may be providing the framework for value to be struck by others.
However, while agreeing with both of these good points that you make, I think that there are still other situations where the people commissioning the evaluation do not want the evaluator to either think about striking a value themselves, or to provide a framework for others to strike a value – they just want them to figure out if a program caused something to happen – and are not really interested in the evaluator’s or the people the evaluator interviews views about value. And while some may not want to call that evaluation, at the moment I am not prepared to go so far as to say that we should not call that evaluation.
Thanks again for your comments.
Paul
An interesting discussion, and well done Paul for asking a question that needed to be asked.
I would take the argument one stage further: if project- or programme- design is carried out rigourously enough, there will be a full results framework, and the value judgments will ALREADY have been made during the design stage, with the outcomes, goals, objectives etc. defined in the light of these value-judgments. The role of the evaluator will then be simply to assess the extent to which intended results were achieved, and, if applicable, to note any aspects in which there were weaknesses in the logic of the framework used.
I never feel comfortable seeing value-judgment phrases in evaluations (“It was a good project”, “It was well-designed”, etc.). We should be specific about the strengths and weaknesses found, and evaluate them without any reference to value (“it achieved the intended results”, “the logical framework was sound”, etc.). Indeed, it would be interesting to do a survey of evaluation reports, and my guess is that there will be a negative coorelation between the rigour (results-based focus) of the design, and the number of times value judgments appear in the evaluation report.
The value judgment should be made later, by the stakeholders, on the basis of their obervations and on the (factual) findings given in the evaluation report.
Regards
Chris
miniAID.com
Your post is very interesting Chris and pushing the argument even further.
I am thinking that part of this whole thing may be that much of the evaluation literature has tended to come from the direction of people evaulating individual programs where their brief is broad and somewhat open. (e.g. in the case of Goal-Free evaluation there is the notion that the evaluator can cast around and decided what are the ‘true’ goals of the program, that is a lot of discretion!). Whereas many of us working evaluators are so embedded with larger systems such as you describe that we are just doing a very small piece of the work.
I also think that the word ‘value’ has so many overlays of meaning and that it has such a long history within social science – independent, but overlapping with evaluation, that it all gets really complicated when we talk about it.
I don’t want to be arguing that values are not important in various ways when doing any evaluation, it is just that I feel, as you do Chris, that a lot of the talk about valuing in evaluation does not always hit the mark in particular projects (this is not to say that sometimes the approaches and methods around such valuing are exactly what is needed in evaluation).
Thanks
Paul
Dear Paul,
This is food for thought, especially in the context of the growth of Evaluation as a ‘field’ in itself. In the past and (still to a large extent),evaluators came from and were selected on the basis of their own substantive analytical work in the subject matter of the subject and issues that needed to be evaluated. In that case, there is some expectation that ‘value’ produced and its various dimensions will be subjected to some analysis in drawing conclusions and recommendations.
However, as evaluation becomes more of a technocratic process, then the assessment of value is best left to those who are running the programs. I am new to the latter way of running evaluations, and in the one case I am currently involved in, it is more in the role of a mentor — assisting the agency itself to develop its evaluation -a la Patton and a Use-focused evaluation technique. Here all judgements key issues and definitions/assessment of value belong with the ‘Users of the evaluation’ who are the primary stakeholders.
The dichotomy I make may seem a bit overdrawn, as there are significant areas of overlap, but I do it to make the point, and to generally — agree with what you are pointing to.
Ms. Shubh Kumar-Range, Ph.D.
Community of Evaluators, South Asia
Dear Shubh,
Yes that is a very good point. So there are evaluations where the evaluator is an expert in an area ans is expected to bring the expertise to working out whether or not the program is of value.
Then there are ones where the role of the evaluator is to provide a framework for others to determine the value of the program (like it sound you are doing in the program you refer to).
Then there are ones where the evaluator even though they are not an expert in the field, develops the framework and does some valuing themselves.
Lastly there are the ones I was referring to in my original post where the evaluator is just being asked to asked to measure whether something has happened and whether it is attributable to the program or to other factors and there is not a lot of discretion for them to get into the business of ‘valuing’.
Thanks for your comment
Paul Duignan
Hello Paul,
It seems your argument holds water. As others have mentioned, whether evaluators determine the value of a program (which should not be confused with whether evaluators make statements that are value-based) depends greatly on the purpose or goal of the evaluation. In other words, evaluators may be hired to determine the value of a program, i.e. whether it is “worth it or not.” However, it seems that you are referring to program-goal evaluations, yes? By program-goal evaluations I mean evaluations conducted to determine the extent to which program goals have been met (and of course the extent to which the program itself is responsible for the results). If this is what you are referring to, then I believe you are correct, i.e. evaluators should not be determining the value of a program. This is especially the case, in my opinion, for programs funded with tax dollars. If the political process has already determined that the program is valuable and needed, then it is not the evaluators place to answer that question. The evaluator should be providing information that will assist in the determination of whether the program is of value, which may be “add-on” after determining whether the program goal has been realized.. Although we may be unhappy with the political process through which public programs are created, that is another issue altogether, and not the evaluator’s role. It seems the evaluator might have a responsibility to also report information that demonstrates the program’s value (or lack of value), but that should be an aside to the main goal of the evaluation, which is to determine whether the program goal was achieved, and to what extent it was achieved (and to show causation where possible), once again assuming that we are talking about program-goal evaluations.
I know you were looking for opposing feedback, but perhaps it is also beneficial to know that others opine as you do.
Hello All,
following on Paul’s little ‘v’ and implicitly a big ‘V’ comment, I am of the opinion that the ‘value’ in Evaluation is a broad concept and all embracing, taking on all the ‘v’s. It takes on everything to do with conclusive statements made about the performance of a program. Whether formative or summative, some judgment is made about the program. Some of these judgments may be considered small ‘v’s while others may be seen as big ‘V’s. I opine that conclusive statements like ” the program is being implemented according to plan” or “the program reached the target group” or “the program had an impact on the community” are all part of this broad conception of ‘valuing’ in evaluation. In conclusion, the evaluator ought to play a role (central or facilitator or whatever) in determining this broad ‘value’. The formative/process or small ‘v’s may have no problem with the evaluator as valuing agent, though the big ‘V’s may need negotiation with stakeholders. I contend that irrespective of the ‘v’ and how it is determined or the role of the evaluator, s(he) needs to do something more than simply research to qualify the process an evaluation.