Making outcomes theory more concrete – checklist for assessing outcomes systems April 21, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Strategic planning, Using the approach , 1 comment so farMost normal people would think that it’s very very obscure, but I’ve just put up a Checklist for Analyzing Outcomes Systems in the Outcomes Theory Knowledge Base and it’s a very exciting development. Up until now the Outcomes Theory Knowledge Base has consisted of a set of articles which outline various aspects of outcomes theory. Outcomes theory is a general theory which covers all types of outcomes systems. Outcomes systems are any type of performance management system, results-base system, monitoring system, evaluation system, outcomes-focused contracting system, or strategic planning system (the term even includes evidence-based practice systems). Such systems have, in the past, been seen as somewhat different types of things without a common theory existing to analyze them. Outcomes theory is based on the insight that we can theorize them as a common type of system and then use the theory to work out how such systems should be best structured. This approach becomes powerful at the moment that we can start applying it to actual real-world outcomes systems. This is the role of the checklist I’ve just developed. (more…)
Problems in pay for performance systems April 9, 2009
Posted by Paul Duignan in : Accountability, Attribution, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , 2commentsSorry, I stopped blogging there for a day or two due to a computer problem, and I will also not be blogging over the Easter Break, but will be back daily blogging after that. Today I’m looking at problems in a pay for performance system. From the point of view of outcomes theory, pay for performance systems are just another example of an outcomes system. Outcomes systems are any system which attempts to identify, measure, attribute and hold people, organizations or programs to account. A U.S. GAO report [2] on one such system is interesting reading (for those with a taste for obscure government reports). It reviews the National Security Personnel System which has just been put on hold by the Obama administration and they may axe it [1]. The problems identified in the GAO report include: (more…)
Why just about every indicator system in the world needs to be fixed! April 5, 2009
Posted by Paul Duignan in : Accountability, Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Standards, Using the approach , 1 comment so farI’ve just posted a new article in the Outcomes Theory Knowledge Base on why it is essential to map indicators onto an underlying visual outcomes model. I blogged a little while ago about why we should be wary of too-tidy indicator sets and in the article I explain why. The vast majority of indicator systems in the world suffer from the problem set out in the article – they are just a straight list of indicators set out in tabular format. They give the user no idea as to whether a number of important steps and outcomes are not being measured. Those using such systems remain blissfully unaware of this. In my view, all these straight indicator sets need to be fixed. It’s not particularly difficult, it just requires some work. How to draw the underlying outcomes models is set out in the outcomes model standards and how to then use such models for indicator mapping and many other things is described in detail in the applied version of outcomes theory – Easy Outcomes. (more…)
Using visual outcomes models to describe and communicate best practice April 3, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Evaluation planning, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Research influening policy, Strategic planning, Using the approach , 4commentsYesterday I blogged about what is meant by the term ‘best practice’. As I said then, I think that there is some conceptual confusion out there about it, and I am not sure that I have yet tidied it up my own thinking about it in a satisfactory way. However, the great thing is that regardless of how we define it, the idea of identifying the types of things that people currently call best practice and communicating these between programs is a great idea. The most difficult thing in many cases is to get best practice actually applied to on the ground after we have identified it. I have put up an Outcomes Theory Knowledge Base article (link to the article is at the bottom of this posting) on the issue suggesting that an efficient way of describing and communicating best practice may be to use visual outcomes models (a type of logic model). (more…)
Simplifying outcomes terminology – angels dancing on the head of a pin! April 1, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes theory, Reporting systems, Using the approach , add a commentIn the middle ages, churchmen occupied themselves by having long discussions sitting around the fire drinking fine wine and discussing how many angels could dance on the head of a pin. It was tough, but someone had to do it! That was until someone called William of Occam came along. He is famous for spoiling the party with something called ‘Occam’s Razor’ – one way of describing the way he operated was that the did not want people to ‘multiply entities beyond necessity’ (see Wikipedia entry). What he was talking about was trying to keep things as simple as possible. This is the approach which is taken within outcomes theory. (more…)
Tables versus visual models March 19, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Outcomes models, Reporting systems, Using the approach , 3commentsEach day this week I am blogging on some of the themes that came up at a national philanthropy conference earlier in the week where I presented on the Easy Outcomes approach. There was some discussion in the conference session about the dynamic nature of what grantees do and how it is hard to capture such dynamism in any rigid system of evaluation. However, as is often the case, at the end of the day, some tabular logic models were presented as an example of how to set out an evaluation. I am not criticizing their presentation, because this is currently standard practice in evaluation planning, and the use of some sort of logic model is always much better than not having one. However, in my view, we fall back on the use of tables because we do not realize that there are better ‘technologies of representation’ (i.e. visual outcomes models) that we could use.
(more…)
Beware of suspiciously tidy indicator sets March 19, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Easy Outcomes, Indicators, Measurement, Outcomes models, Reporting systems , 1 comment so farI’ve just come away from presenting at a national Philanthropy conference as part of a half day session on evaluation and outcomes. I was presenting on the use of the Easy Outcomes approach as a way of grantees structuring outcomes, indicators and evaluation. I will tidy up the outcomes model I used and post a link to it in a blog in a week or so. A lot of interesting points came up in the discussion and I will blog on several of them over the next few days. The first one is to be beware of suspiciously tidy indicator sets. The Easy Outcomes approach gets people to draw an outcomes model (intervention logic) of what they are trying to do without worrying about what they can and can’t measure and what they can, and can’t demonstrate is attributable to their particular project (both of these issues are dealt with later in the process). You draw the models using the guidelines here.
(more…)
Bonuses role in the financial melt-down March 16, 2009
Posted by Paul Duignan in : Accountability, Indicators, Measurement, Outcomes models, Outcomes theory, Outcomes theory & politics, Outcomes theory & the news, Reporting systems , add a commentPresident Obama has amplified the attack on bonuses being paid to staff in companies which have been bailed out by the U.S. government (CNN, 16 March 2009). What does outcomes theory have to say about the role of the bonus system in the current financial meltdown? I blogged in 2007 about the problem of bonuses in the financial system and how it was possible the full extent of the melt-down would take time to be revealled. Thinking in terms of an outcomes model, what has happened is that financial institutions should have been aiming (as they do in healthy times) at the overall outcome of Sustainable long-run profitability. (more…)
Distinguishing Evaluation from Monitoring (and other processes such as Performance Management and Assessment) January 30, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Systematic Outcomes Analysis, Using the approach , 5commentsI have blogged before about what I see as the wrong way to approach the problem of differentiating evaluation from related activities (e.g. monitoring, performance management, assessment). Last time it was about differentiating evaluation from quality assurance. The wrong way is to spend a lot of time on attempting to distinguish between processes by using high-level terms such as evaluation, performance management etc. A lot of ink and talk has been spilled on trying to do that over the years. In the case of monitoring and evaluation there is a simple distinction which can be drawn between monitoring being about routinely collected information and evaluation being about more ‘one-off’ studies. This distinction is useful sometimes, however it is not much good for any more sophisticated discussion of differences in evaluation, monitoring, performance management and related processes which people use in particular situations.
A more useful approach is set out in an article I have just written on Distinguishing Evaluation from Other Processes. The problems with trying to differentiate high-level terms are: 1) that we cannot control the way that stakeholders use them in the field; and, 2) these high-level terms for processes are not fully mutually exclusive, that is, sometimes the same activities are undertaken within processes even though the processes are thought to be distinct (e.g. evaluation, performance management, etc.). (more…)
Developing a comprehensive sector visual outcomes model January 20, 2009
Posted by Paul Duignan in : Communicating outcomes models, DoView, Easy Outcomes, Indicators, Outcomes models, Reporting systems, Uncategorized , 2comments
In earlier blog postings I’ve talked about the use of large scale outcomes models for various purposes such as overall policy development, evidence-based practice and monitoring and evaluation. The next version of DoView outcomes and evaluation software [[Disclosure: I am involved in the development of DoView] is going to allow images to be included in DoView files and in the web page models which can be created within DoView and then put up on the web. A mock-up of what parts of a visual sector plan using the upcoming version of DoView could look like is available. This will be a no cost update to DoView which is planned for release in February 2009.