If all knowledge is subjective, why listen to evaluators? March 24, 2009
Posted by Paul Duignan in : Outcomes theory & politics, Philosophy of science , trackbackThis week I’m blogging on a Rich Dialog Process (RDP) I facilitated recently on the interface between research (and evaluation) and policy. Check out my last couple of blog postings for more information on the actual process. One of the most interesting moments for me was a relatively short interchange around the issue of research ‘agendas’ and subjectivity and objectivity in research. I decided that given the range of issues we had to cover in the RDP there was not enough time to get right into this topic, important though it is, and the participants did not seem to want to get into it at that moment, so we moved on. However, this is an issue which I’ve been interested in for a long time.
One day I would love to have the opportunity to sit down with a group of researchers/evaluators with different perspectives and really explore it with them – it may well end up in a screaming match – but I’m a clinical psychologist in addition to an evaluator and we tend to enjoy refereeing that sort of thing! Also a lot of words have been spilt on this issue in the evaluation literature for a long time. To put the issue in one way – and no doubt a crude way that will offend some people holding some positions on the issue – if all knowledge is subjective (as someone said in a brief comment in the RDP) then why should policy makers or anyone else put extra weight on what researchers have to say over and above the myriad of advocates who come to them spouting data and analysis on one topic after another?
The idea that ‘all knowledge is subjective’ now has a tight hold on certain parts of the social science imagination, there is a long history to this which I would like to write about some day when I get a moment. Illustrative of the hold it has is that in some classes of postgraduate evaluation students, when I ask them to put up their hands if they believe that ‘all knowledge is subjective’, I get a large majority who think that it is. Of course, like any catch-cry the phrase ‘all knowledge is subjective’ needs to be unpacked and nuanced to truly capture the positions they will be holding.
I think that it is easier for general social science researchers who are not doing particularly policy relevant research to hold to a notion such as this because they are not so much in the firing line as evaluators and researchers who have to do policy relevant research. The problem for an evaluator is that they are often in the business of making statements about whether programs work or not. These statements have implications for people and their jobs in the real world.
One of the first evaluations I worked on consisted of a request that I edit and tidy up an evaluation report. I did so and the report was subsequently released. We were immediate threatened with a defamation case by one of the parties mentioned in the report. Fortunately it came to nothing in the end, however this was a good early lesson for me as an evaluator. This type of experience tends to focus the mind on the question of what we are saying as evaluators and, to put it in researcher-speak, the ‘epistemological status of our knowledge claims’.
I had a similar ‘epistemological moment’ in my early work as a clinical psychologist. I was working in a child and family clinic. Some psychotherapists – not clinical psychologists – with something of a psychoanalytic bent – were involved in trying to identify whether or not children had been sexually abused. I wasn’t involved in the cases, but I used to read their reports, as part of my training, with amazement. They seemed to me to be making the most dramatic claims about possible abuse with the flimsiest of impressionistic evidence presented in their reports. I thought at the time that that type of analysis was not going to fly the moment it hit the court process. Subsequently controversies around this issue have reflected this.
So people who make claims which affect other people need to be clear of the basis on which they make those claims. Part of the problem for social science has been the great wave of post-modernism and its accompanying relativism which swept through the social science disciplines over 1980s and 1990s. What we are seeing in the social sciences, and this emerges in a lot in debates within evaluation, is a sort of rubbing up of tectonic plates (where I live – Wellington, New Zealand – sits directly on a earthquake fault line so I know all about these) between the claim that all knowledge is subjective and the more and more strident demand for ‘hard evidence’ with which to inform the policy debate. A particular requirement of the ‘hard evidence’ and why there are many intelligent people of various political pursuasions calling for it is that it somehow sits on a foundation which is different from all of the ‘selective evidence’ and ‘spin’ which is fed into the political process day after day. Any intelligent person looking at the political decision making process today increasingly sees it as simply a matter of interested parties throwing enough money at an issue to swing the decision in a way that benefits them economically and makes a hansome return on the investment they have put into spin.
It is easy to take one or other of the sides in the overall issue. One side is to just say that ‘all knowledge is subjective’ and to leave it at that. This is best done if you are buried away in some part of social science within a university where you are not likely to be on the receiving end of too many defamation claims. The other side is to just claim that ‘scientific research provides objective information on the way the world works, end of story.’
I spent a lot of time at one stage reading philosophy of science to try and bridge this gap, in fact in my doctorate draft I had a long section which basically attempted to summarize the history of Western philosophical thought on this issue. Fortunately my supervisor Dr John F Smith, wisely convinced me to cut all of this out. For better, or worse, the place where I ended up, in terms of philosophy of science, was what I call pragmatic post-modernism, which drew on the work of the philosopher Richard Rorty. (Here is the Wikipedia entry for him, the photography of him up there somewhat daunting at the moment – I preferred him as he appeared on one of his paperbacks – I think Contingency, Irony and Solidarity, strutting around in a white suit, a little bit like Tom Wolfe – but I digress). I liked pragmatic post-modernism because it seemed to me that it allowed you to sort of import lots of pragmatic methodologies (like we use in science, research and evaluation) into a framework where it was not claimed that they were true – because making such claims was basically an empty gesture.
The claim was simply that it was possible to sit down with a group of people who are interested in what action should be taken in the future and work out a set of criteria for deciding on what that action should be. Taking this approach, the general argument is that if people are wanting to solve a social problem, they may well agree that it would be a good idea to have a group of people (researchers or evaluators) to look at the issue analytically and to collect data on that issue and to subject each other to a process of critical peer review so that the ideas are related to data on the issue and arbitrary claims which are not reflected in the data are winnowed out in the process.
Anyway, this has turned into an extended blog, as you can see from the above, I think that the issue of subjectivity and objectivity in research and evaluation is a really important one which, in spite of all that has been written about it, we need to grapple with again and again because the tensions which lie under it are not going to go away – particularly the desire of people for a way out of the current spin-controlled decision-making which is now clearly being revealed as not being a really coherent way of making society-level decisions about our societies (take the economic melt-down or climate change as current major examples).
Paul Duignan, PhD
Comments»
[...] the original post: If all knowledge is subjective, why listen to evaluators? Share and [...]