Impact evaluation’s day in the sun (Part II) December 1, 2008
Posted by Paul Duignan in : Attribution, Impact evaluation, Measurement, Uncategorized , trackback
(See the previous posting Part I first). Progressing the debate around impact evaluation means unpacking some of the different issues which are involved in the debate. In doing this, it should not be assumed that those on various sides in the debate will always side in the same way with each of the sub-issues. In my doctorate (Duignan, 1997) I did this type of analysis for the positivist versus constructivist debate which was hot for a while within evaluation circles. Drawing on that analysis, we can now to look at the sub-issues which underly the impact evaluation debate because there are a number of similarities with the earlier debate. The sub-issues which can underly the impact evaluation debate and two contrasting positions that can be taken on them include the following:
- Epistemology - We can develop objective knowledge of the world versus Knowledge is subjective
- Control of evaluation research agenda - Bureaucratic/elite/First World control versus Grass roots/Third World involvement in research agenda
- Participation of those studied - People studied treated as similar to objects versus People studied consulted, given control and given feedback
- Methods - Quantitative methods versus Qualitative methods
- Number of variables - It is possible to study tow or so variables in isolation by controlling for other variables versus Variables can only be studied as part of a total systems interaction
- Generalizability - Findings from one study can be generalized to other situations and general laws developed versus Findings are closely tied to particular circumstances, when they are used elsewhere it is always an extrapolation
- Rationality and society - Scientific rationality is a major way to move towards a better society versus Scientific rationality has caused a divided, polluted, unsustainable and inequitable world
- Values - Evaluators should not let their values influence their question framing, methods or interpretation versus Evaluators cannot help their values influencing their question framing, methods and interpretation
- Facts/truth and interpretation/meaning - Evaluators should seek facts/truths versus Evaluators should seek interpretations and meanings
- Power - Despite power imbalances truth will out in the end versus Science is used by the powerful to support their world views
- Unified science - The same underlying approaches can be used for the study of physical and social worlds versus Studying people and society requires a totally different approach from studying objects
- Exercise of authority - Scientific (evaluation) discourse is a power neutral description of truth versus Scientific discourse is inextricably bound up with the exercise of power
- Right to one’s own representations of the world - There is only one valid representation of the world versus There are many representations of the world and oppressed people have a right to develop, control and voice their own representations.
This means that when people think that they’re having the impact evaluation debate the different parties may be having a range of debates in their heads or in person about the different sub-issues which I have set out above. For instance a colleague raised with me the fact that some on the impact evaluation side of the debate in international development were saying that ‘doner fatigue’ had set in. Now my colleague was no doubt thinking along the lines that this statement was a neo-colonial sentiment and no wonder aid was not working because it was too little too late in a world which is structured to advantage the First World against the Third World. Those making this type of statement may or may not be thinking similar thoughts but if aid is to make a difference impact evaluation evidence is going to be one of the ways the point can be made back to the First World. This is an example of how a sub-issue, obviously an important one to discuss, was coming to the fore in the impact evaluation debate. It is important to repeat the point I made at the top that the side of these sub-issues which people come down on may vary among those who support more impact evaluation being done, it should not be assumed that there are two sets of positions on these sub-issues which always line up in the same way for any individual. Now, of course, there are a lot of issues here to work out and if I am proposing that we sort all of these out before we make any progress on the impact evaluation debate I would be dreaming. What we need to do is to figure out a sensible way forward, but being aware that there are a lot of sub-issues under this debate is a step in the right direction because it helps to clarify why so many people feel so much frustration at the debate and the way it tends to go at the moment. In my next posting tomorrow I will outline how I think we should move forward with the debate (Part III to come).Duignan (1997) Evaluating health promotion: The strategic evaluation framework. University of Waikato. p. 141-147. Paul Duignan(OutcomesBlog.org)
Comments»
[…] Read Part I and Part II first]. Now that I have got to Part III of this posting on the impact evaluation debate, the time […]