A Systematic Outcomes Analysis framework for psychotherapy evaluation February 12, 2008
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes systems architecture, Systematic Outcomes Analysis, Outcomes models, Easy Outcomes, Evaluation planning, DoView , trackback
In my last blog posting (which you should read before this one) I talked about using Systematic Outcomes Analysis to define the basic tasks one needs to do in quality assurance, monitoring and evaluation and how this can avoid the need for a protracted theoretical discussion about the difference between quality assurance and program evaluation. I was using the example of an illustrative Systematic Outcomes Analysis framework I set up based on an outcomes logic model in regard to psychotherapy which I’ve posted on the Outcomes Models site. Here’s the PDF of the DoView file. Using the Systematic Outcomes Analysis approach, indicators and evaluation questions are mapped onto the outcomes logic model (indicators are marked with a yellow icon and evaluation questions with a green circular icon). This blog posting looks in more detail at ways stakeholders can use such a framework once it’s been developed.
Before I jump right into this, I’ll just make one quick point of explanation to help with the discussion below. Systematic Outcomes Analysis identifies five major features of steps or outcomes in an outcomes model. They can be: influencable - able to be influenced by a player; controllable - only influenced by one particular player; measurable - able to be measured; attributable - able to be attributed to one particular player (i.e. proved that only one particular player changed it); and accountable - something that a particular player will be rewarded or punished for.
Once a framework like the one for psychotherapy being discussed here has been set up, it can be used by stakeholders in various ways. The key diagram (slice) to refer to is the one at the top of this blog posting which sets out the outcomes model, indicators and evaluation questions. Some possible ways this framework could be used by stakeholders are:
- Stakeholders can discuss who’s going to be responsible for doing which of the tasks set out in the analysis. Some of the indicators will be routinely monitored and reported on by the psychotherapists themselves; they may also agree to answer some of the process evaluation questions or formative evaluation questions about improving aspects of current practice. Stakeholder also need to consider whether or not they will attempt to answer the high-level outcome evaluation question in the case of a particular group of psychotherapists. Often it’s decided that it’s only feasible and affordable to monitor the lower levels of an outcomes model. In these cases reliance is placed on previous research as having established that there’s a reasonable likelihood that if the lower level steps in the model are achieved (e.g. the right techniques applied well) that higher-level outcomes will flow from these. (See the Easy Outcomes site for more discussion of the two possible monitoring and evaluation schemes).
- Psychotherapists and their funders can discuss which indicators they agree the psychotherapists should be held accountable for - i.e. rewarded or punished for. This first requires a decision as to which type of accountability contracting the psychotherapists are working under - Systematic Outcomes Analysis identifies three possible accountability contracting arrangements - contracting for attributable indicators (outputs) only, contracting for attributable indicators (outputs) plus ‘managing for outcomes’ , or contracting for not-fully controllable outcomes. (See the Easy Outcomes site for more discussion of these). Once they’ve made this decision, they can then identify the particular indicators the psychotherapists will be held to account for. If they’re only contracting for attributable indicators (outputs) then they’ll only be able to contract for attributable indicators - these indicators are marked with an A.
- Attempts could be made to make some of the indicators ‘more attributable’. For instance, the psychotherapists may argue that the length of time they see a client (and hence the cost) is not attributable to them because it’s not fully controllable by them and hence it should not be one of their accountable indicators. This is because different psychotherapists may have client groups which need differing lengths of therapy. A method such as casemix may be able to make an indicator ‘more attributable’ so to speak. The casemix approach undertakes research on a large number of psychotherapists to identify groups of patients who on average take a certain length of time (and hence cost) to treat. Once the average figures for treating such clients have been established, an individual psychotherapist’s (or a group of psychotherapists’) length of treatment (or cost) indicators are adjusted to allow for the particular mix of clients they’re seeing. It can be argued that such an ‘adjusted’ indicator is more attributable to a psychotherapist or a group of psychotherapists than the raw indicator measuring just the average length of therapy.
So, the purpose of having a Systematic Outcomes Analysis like this for stakeholders to work from is to ensure that the discussion about who is doing what and who should be accountable for what proceeds in an orderly manner. Being able to show the whole picture like this including outcomes model, indicators and evaluation questions keeps such discussions on track. I’ve had far too many confused discussions about quality assurance, monitoring and evaluation where it gradually becomes clear that the different people who’re arguing about these issues all have somewhat different understandings of the underlying tasks they’re talking about and trying to get done.
Even if stakeholders are not attempting to undertake a particular task, e.g. outcome evaluation, it’s helpful to still have the outcome evaluation question visualized in front of them when discussing what they do and don’t want done. Using this kind of comprehensive framework in a case like this means that everyone can agree that they’re NOT talking about answering the high-level outcome evaluation, and this type of conceptual precison often helps the clarity of stakeholder discussions. In my experience, the work required to set up a framework like this is more than compensated for by the transparency which ensures in later stakeholder discussions.
Paul Duignan (outcomesblog.org)
Comments»
[…] my next blog posting I’ll discuss in more detail how stakeholders can use a framework like this once it’s […]
I am currently examining the possibility of creating a program evaluation for a very specific group: individuals who are experiencing symptoms consistent to post-traumatic stress disorder. There are many possible ways someone could experience traumatic stress, but the most researched pathways are combat-related stress/trauma and trauma following sexual assault. In working to create an appropriate evaluation, and possible stakeholders, I am trying to decide whether it would matter if participants in my group were solely military personnel, or if it would be permissible to have a group composed of military persons and civilians. The trauma exposure would be very different; the diagnosis would be the same.
At the heart of my question: which is more important in looking at psychotherapy from a program evaluation lens: given the program is a group, not an individual, is it more important to focus on the diagnosis (PTSD, where anyone with the diagnosis would be included regardless of the trauma source), or the group participant identity (where it may be preferable to have a more homogeneous group of military personnel returning from armed conflict, as opposed to a heterogeneous group who have been diagnosed with PTSD as a result of exposure to trauma of any kind). Thank you.