Problems in pay for performance systems April 9, 2009
Posted by Paul Duignan in : Accountability, Attribution, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , 2commentsSorry, I stopped blogging there for a day or two due to a computer problem, and I will also not be blogging over the Easter Break, but will be back daily blogging after that. Today I’m looking at problems in a pay for performance system. From the point of view of outcomes theory, pay for performance systems are just another example of an outcomes system. Outcomes systems are any system which attempts to identify, measure, attribute and hold people, organizations or programs to account. A U.S. GAO report [2] on one such system is interesting reading (for those with a taste for obscure government reports). It reviews the National Security Personnel System which has just been put on hold by the Obama administration and they may axe it [1]. The problems identified in the GAO report include: (more…)
Social Innovation, evaluation and outcomes April 6, 2009
Posted by Paul Duignan in : DoView, Easy Outcomes, Impact evaluation, Outcomes models, Outcomes theory, Outcomes theory & politics, Research influening policy, Using the approach , add a commentI attended a launch of the New Zealand national Center for Social Innovation last night. Geoff Mulgan from the Young Foundation (a similar center in the U.K.) talked about social innovation. The social innovation movement is about getting stakeholders and sectors together to do things differently to achieve better social outcomes. Already a dynamic movement, it has recently received a shot in the arm from the global economic melt-down – traditional ways of doing things are increasingly being questioned and people are looking for new solutions. A number of points made by Geoff and in the subsequent discussion are particularly relevant to outcomes and evaluation were: (more…)
Using visual outcomes models to describe and communicate best practice April 3, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Research influening policy, Strategic planning, Using the approach , 4commentsYesterday I blogged about what is meant by the term ‘best practice’. As I said then, I think that there is some conceptual confusion out there about it, and I am not sure that I have yet tidied it up my own thinking about it in a satisfactory way. However, the great thing is that regardless of how we define it, the idea of identifying the types of things that people currently call best practice and communicating these between programs is a great idea. The most difficult thing in many cases is to get best practice actually applied to on the ground after we have identified it. I have put up an Outcomes Theory Knowledge Base article (link to the article is at the bottom of this posting) on the issue suggesting that an efficient way of describing and communicating best practice may be to use visual outcomes models (a type of logic model). (more…)
What exactly is ‘best practice’? April 2, 2009
Posted by Paul Duignan in : Evaluation debates, Impact evaluation, Outcomes theory, Philosophy of science, Use of terms, Using the approach , add a commentIdentifying and communicating best practice is widely recommended in many sectors and disciplines. But I’ve sometimes wondered as I’ve sagely recommended in a serious voice, ‘I think that we should use an approach based on identifying and implementing best practice here’ exactly what best practice is? I think that doing it is often a good idea and I can work out how to identify it and share it, and I will blog about that tomorrow, but what I’m not clear on is exactly how we define ‘best’ in the term ‘best practice’. It’s not clear whether best practice consists of: 1) claims that practitioners, from their own experience, believe the practices concerned to be feasible and ‘useful’ to implement; or 2) practices which have been proven to improve high-level outcomes (through making a strong outcome/impact evaluation claim of some sort such as is made using some of the types of designs listed here). (more…)
Simplifying outcomes terminology – angels dancing on the head of a pin! April 1, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes theory, Reporting systems, Using the approach , add a commentIn the middle ages, churchmen occupied themselves by having long discussions sitting around the fire drinking fine wine and discussing how many angels could dance on the head of a pin. It was tough, but someone had to do it! That was until someone called William of Occam came along. He is famous for spoiling the party with something called ‘Occam’s Razor’ – one way of describing the way he operated was that the did not want people to ‘multiply entities beyond necessity’ (see Wikipedia entry). What he was talking about was trying to keep things as simple as possible. This is the approach which is taken within outcomes theory. (more…)
Extraordinary circumstances and Dick Cheney’s ‘stuff happens’ March 29, 2009
Posted by Paul Duignan in : Accountability, Attribution, Evaluation debates, Outcomes theory, Outcomes theory & politics, Outcomes theory & the news , add a commentIn my last blog posting I commented on Jon Stewart’s critique of ex-Vice President Dick Cheney’s claim that the Bush administration should not be held accountable for the U.S. economic melt-down because a number of things happened during their term which affected the economy. The Vice President summarized this by saying that ‘stuff happens’ and this ‘stuff’ unexpectedly blew their budget. The ‘stuff’ included the wars in Afghanistan and Iraq, and Hurricane Katrina. In technical outcomes theory terms, the Vice-President was mounting an ‘extraordinary factors’ argument to reduce his administration’s accountability for the economic melt-down. (more…)
To attribute or not to attribute – Jon Stewart vs Dick Cheney March 27, 2009
Posted by Paul Duignan in : Accountability, Attribution, Evaluation debates, Outcomes theory, Outcomes theory & politics, Outcomes theory & the news , 3commentsIn a recent episode of Jon Stewart’s Daily Show he deconstructs a high-profile interview with Dick Cheney, the previous Vice President of the United States, undertaken by another interviewer. While this is a comedy show, being an overly analytical sort of person, I can’t watch it without analyzing what is going on it in from the point of view of outcomes theory! At a technical level, the key issue Stewart is focusing on in creating his laughs is what in outcomes theory is called – demonstrable attribution. Demonstrable attribution is being able to demonstrate that an improvement which occurs following an intervention has been caused by a particular intervention (see here for more outcomes theory information on this). In summary, Stewart claims that Dick Cheney is applying a double standard around demonstrable attribution. (more…)
Bonuses role in the financial melt-down March 16, 2009
Posted by Paul Duignan in : Accountability, Indicators, Measurement, Outcomes models, Outcomes theory, Outcomes theory & politics, Outcomes theory & the news, Reporting systems , add a commentPresident Obama has amplified the attack on bonuses being paid to staff in companies which have been bailed out by the U.S. government (CNN, 16 March 2009). What does outcomes theory have to say about the role of the bonus system in the current financial meltdown? I blogged in 2007 about the problem of bonuses in the financial system and how it was possible the full extent of the melt-down would take time to be revealled. Thinking in terms of an outcomes model, what has happened is that financial institutions should have been aiming (as they do in healthy times) at the overall outcome of Sustainable long-run profitability. (more…)
What’s in a definition? March 15, 2009
Posted by Paul Duignan in : Blog info, Communicating outcomes models, Outcomes systems architecture, Outcomes theory , 2commentsI have not blogged for a number of weeks due to family illness and a bereavement. However, now I am back on deck and planning to increase the frequency of my blogs – shorter on some occassions and more often, but there will still be some substatial ones for those who want substance. If there are any issues you want me to blog on, or any questions you would like me to answer in future blogs, just post a comment to this posting.First up, I have now added to the Outcomes Theory Knowledge Base a set of definitions as they are used in outcomes theory. (more…)
Distinguishing Evaluation from Monitoring (and other processes such as Performance Management and Assessment) January 30, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Systematic Outcomes Analysis, Using the approach , 5commentsI have blogged before about what I see as the wrong way to approach the problem of differentiating evaluation from related activities (e.g. monitoring, performance management, assessment). Last time it was about differentiating evaluation from quality assurance. The wrong way is to spend a lot of time on attempting to distinguish between processes by using high-level terms such as evaluation, performance management etc. A lot of ink and talk has been spilled on trying to do that over the years. In the case of monitoring and evaluation there is a simple distinction which can be drawn between monitoring being about routinely collected information and evaluation being about more ‘one-off’ studies. This distinction is useful sometimes, however it is not much good for any more sophisticated discussion of differences in evaluation, monitoring, performance management and related processes which people use in particular situations.
A more useful approach is set out in an article I have just written on Distinguishing Evaluation from Other Processes. The problems with trying to differentiate high-level terms are: 1) that we cannot control the way that stakeholders use them in the field; and, 2) these high-level terms for processes are not fully mutually exclusive, that is, sometimes the same activities are undertaken within processes even though the processes are thought to be distinct (e.g. evaluation, performance management, etc.). (more…)