Psychometrics of the ORS and SRS

The ORS and SRS are very brief, feasible measures for tracking client well-being and the quality of the therapeutic alliance, taking less than a minute each for clients to complete and for clinicians to score and interpret. The ORS has been shown to be sensitive to change among those receiving services. Numerous studies have documented concurrent, discriminative, criterion-related and predictive validity, test-retest reliability, and internal-consistency reliability for the ORS and SRS (e.g., Anker et al., 2009; Bringhurst et al., 2006; Campbell & Hemsley, 2009; Duncan et al., 2003; Duncan et al., 2006; Miller et al., 2003; Reese et al., 2009). The significant impact of using these measures on the outcome of services has similarly been well-documented by numerous researchers (e.g., Anker et al., 2009; Miller et al., 2006; Reese, Norsworthy, & Rowlands, 2009).

The Outcome Rating Scale

The Outcome Rating Scale (ORS) is a brief, client rated, four-item visual analogue scale measuring the client's experience of well-being in his or her individual, interpersonal, and social functioning. The ORS is designed and normed for adults and adolescents (ages 13+). The Children's Outcome Rating Scale (CORS) has been normed for ages 6-12. The Young Children's Outcome Rating Scale (YCORS) is a “clinical engagement" tool for children below 6 years. The YCORS is not scored by the clinician; it is used to provide very young children a way of expressing their well-being and satisfaction with a therapy session, as do the older children and/or adults with whom they may be in treatment.

Clinical Cutoff of the ORS

Determining the clinical cutoff for an outcome measure accomplishes two related objectives: (1) It defines the boundary between a normal and clinical range of distress; and (2) it provides a reference point for evaluating the severity of distress for a particular client or client sample. Using the method described by Jacobson and Truax (1991), the clinical cutoff for the ORS was determined to be 25 (Miller, Duncan, Brown, Sparks, & Claud, 2003). The sample on which this score is based is quite large (n=34,790) and comparison with other well-established measures shows it to be a reasonable differentiator between “normal" and “clinical" levels of distress.

For example, the clinical cutoff score for the OQ-45 falls at the 83rd percentile of the nontreatment sample, and the clinical cutoff for the ORS falls at the 77th percentile of the nontreatment sample. Miller and colleagues have reported that between 25-33% of people presenting for treatment score above the clinical cutoff at intake (Miller & Duncan, 2000; Miller, Duncan, Sorrell, & Brown, 2005). While the clinical cutoff for adults is 25, younger clients tend to score themselves higher. Therefore, the clinical cutoff for adolescents (age 13-18) is 28, and for children (age 6-12) the cutoff is 32.

The Reliable Change Index

When treatment is successful, scores on the ORS should increase over time. To be able to attribute such changes to nonrandom, substantial changes in well-being, the difference between any two scores must exceed a statistical index known as the reliable change index (RCI).

Briefly, the RCI indicates change that is very likely to be greater than chance or day-to-day variation in a person's scores (Jacobson, 1988; Jacobson, Folette, & Revenstorf, 1984; Lambert & Hill, 1994). Change that both exceeds the RCI and crosses the clinical cutoff from a clinical to a nonclinical level is called “clinically significant change" (Jacobson & Truax, 1991). With regard to the ORS, the RCI is 5 points (Miller, Duncan, Brown et al., 2003).

Trajectories of Change

In addition to the clinical cutoff, clinicians can receive feedback comparing a client's score on the ORS to a computer-generated “expected treatment response" (ETR) for that session number. As researchers Wampold and Brown (2006) have observed, “Therapists are not cognizant of the trajectory of change of patients seen by therapists in general, that is to say, they have no way of comparing their treatment outcomes with those obtained by other therapists" (p. 9).

Using a large and diverse normative sample that included 300,000 plus administrations of the ORS, Miller et al. (2004) produced algorithms capable of plotting an average trajectory of change over time based on a person's initial score (e.g., level of functioning) on the measure. The resulting graphs resemble and serve a similar function as growth curves used in medicine to assess height, weight, and head circumference.

Having access to individual client trajectories in comparison to the average for a large sample enables clinicians to quickly identify those at increased risk for a null or negative outcome so the clinician and client can alter, augment, or refer for other services before the client drops out of treatment or spends too much time in a direction of treatment that does not optimize the chance of treatment success.

The Session Rating Scale

The Session Rating Scale (SRS) is a four-item, client completed therapeutic-alliance measure. Like the ORS, the SRS is a visual analogue scale that takes less than a minute to administer, score, and interpret. Items on the scale reflect the classical definition of the alliance first stated by Bordin (1979).

The scale assesses four interacting elements, including the quality of the relational bond, as well as the degree of agreement between the client and therapist on the goals, methods, and overall approach of therapy. The SRS is for ages 13 and up, the CORS (Children's Outcome Rating Scale) is for ages 6-12, and the Young Children's Session Rating Scale is for ages 6 and below. The SRS is also available in a group version, the Group Session Rating Scale (GSRS). The SRS measures are available in over a dozen languages and there is a script available for oral administration.

SRS Alliance Cutoff

The cutoff for an alliance measure is the point at which clinicians should be especially alert to the possibility of a rupture in and potential failure of the working relationship. The alliance cutoff enables clinicians to identify those therapeutic relationships that are at a statistically greater risk for client drop outs or experiencing a negative or null outcome from treatment. On the SRS, a score of 36 or below is considered cause for concern because fewer than 24% of cases score lower than 36 (Miller & Duncan, 2004).

Creating a Culture of Feedback

Soliciting clinically useful feedback from consumers of therapeutic services requires more than administering the ORS and SRS. Clinicians must work at creating an atmosphere where clients feel free to rate their experience of the process and outcome of services:

1. Without fear of retribution.
2. With a hope of having an impact on the nature and quality of services delivered.

Beyond displaying an attitude of openness and receptivity, creating a “culture of feedback" involves spending time to introduce the measures thoughtfully and thoroughly. Providing a rationale for using the tools is critical, as is including a description of how the feedback will be used to guide service delivery (enabling the therapist to catch and repair alliance breaches, to prevent drop out, to correct deviations from optimal treatment experiences, etc.). To elicit accurate and relevant feedback from the client, it is important that the therapist introduces the measures carefully, being extremely clear about the questions being asked on the forms and why the client's answers are important.

It can be useful to let clients know that research on the effectiveness of therapy suggests that their experience of early improvement in their situations and their experience of satisfaction with the services they are receiving from their therapists are of prime importance in achieving a successful outcome. It is therefore important that they understand that the therapist is not going to be offended or go on the defensive in response to any negative feedback. On the contrary, the therapist must genuinely feel, and effectively communicate, that he really needs to know if the client does not feel he or she is being helped or if there is something the client wants to be different about the treatment so that he can be responsive to the client.

Naturally, this attitude on the part of the therapist requires a real belief that the client's view of treatment is paramount in importance for effectively driving the process of therapy, and that a client's concerns or desire for a change in the process deserves to be taken seriously and humbly by the therapist rather than being treated as clinical fodder to be understood within a frame of reference with which the client has just expressed dissatisfaction. It is also important to stress to clients that this is not just another form designed to assess them and decide what diagnostic category they fall into, nor is it just a bureaucratic requirement.

The main purpose of the feedback measures is to help the therapist to stay on track and to avoid doing or saying things that are unhelpful or harmful. It is a way the therapist can demonstrate his or her commitment to being accountable to the client.

Introducing the scales to the client

The following is one example of how to introduce FIT to your clients. Please use the example to inspire you to find your own words when introducing the scales to your clients.

“I work a little differently at my practice than many therapists do. One of my top priorities is to make sure that the clients who come to see me benefit from our work and achieve the results they are wanting. For this reason, it is very important that you are involved in monitoring the progress of services from beginning to end. For this reason I use two brief tools to track your experience of the outcome of our work together and your experience of the way we work – the first scale is one we use in the beginning of the session, the second scale is one we use at the end of the session.

Your feedback is critical in ensuring that our work together is useful to you. One of the things we know that often happens in successful treatment is that the client experiences change early in the treatment. This doesn't mean that we have to stop working together quickly, but actually means that there is a good prognosis that our work is going to have a positive effect in the long-term, so the early change is telling us that we are probably on track. If our work is successful and useful to you, we can continue as long as it makes sense to you. If you don't experience an early change, then I'm likely to talk with you about whether we should try to change or modify the services. If things still don't improve, then I may bring up whether we should try something more dramatically different, including perhaps trying to find someone or someplace else for you to get the help you want. Does this make sense to you?"

Superior Therapists and the Culture of Feedback

There is growing evidence that the process of responding to a client's negative feedback, even about an aspect of therapy that may seem relatively trivial, can contribute to the strength of the therapeutic alliance and set in place a strong foundation for future work. There is also evidence that better therapists elicit more negative feedback from their clients.

This suggests that these therapists are able to forge a strong enough alliance with their clients that the clients feel safe in giving them honest feedback. It also illustrates that in building a culture of feedback it is important that the therapist recognizes and believes in its importance. In summary the quality and usefulness of the feedback therapists get from clients will depend on the degree to which the therapist genuinely wants honest feedback and the extent to which this is communicated effectively to the client.

---