slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
What’s New in Outcome Measures? PowerPoint Presentation
Download Presentation
What’s New in Outcome Measures?

Loading in 2 Seconds...

play fullscreen
1 / 75

What’s New in Outcome Measures? - PowerPoint PPT Presentation


  • 364 Views
  • Uploaded on

UKABIF Annual Conference Wednesday 11th November 2009. What’s New in Outcome Measures?. Professor Nick Alderman Consultant Clinical Neuropsychologist St Andrew’s Healthcare Northampton UK. nalderman@standrew.co.uk. Special Issue of Neuropsychological Rehabilitation

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'What’s New in Outcome Measures?' - clodia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

UKABIF Annual Conference

Wednesday 11th November 2009

What’s New in Outcome Measures?

Professor Nick Alderman

Consultant Clinical Neuropsychologist

St Andrew’s Healthcare

Northampton UK

nalderman@standrew.co.uk

slide2

Special Issue of

Neuropsychological Rehabilitation

‘Evaluation of Outcomes in Brain Injury

Rehabilitation’ (1999)

Concluded there is

“...no single solution to a complex problem”

(Fleminger & Powell)

slide4

A Compendium of Tests, Scales and Questionnaires

The Practitioner’s Guide to Measuring Outcomes after Acquired Brain Impairment

By Prof Robyn Tate (University of Sydney)

A comprehensive reference manual providing a detailed review of approximately 150 specialist instruments for the measurement of signs and symptoms commonly encountered in neurological conditions, both progressive and non-progressive.

Published by Psychology Press 6th January 2010

slide5

Structure of Presentation

  • What outcomes to measure?
  • At what level?
  • The Outcome ‘Basket’
  • Challenges to measuring service level outcomes
  • Some ongoing examples of developments in the measurement of neurobehavioural outcomes
  • Some final thoughts
slide6

Service

Users

Outcome

?

Clinicians

Families

Purchasers

slide7

What Outcomes?

Depends on who is asking about what…

Turner-Stokes (1999) suggests:

  • Medics – impairment
  • Therapists – disability and independence
  • Patients & Families – reduction in handicap and quality of life

Rehabilitation – regaining functional independence

Most outcome tools therefore measure:

  • Increase in independence
  • Reduction of disability
slide8

Lynne Turner-Stokes (1999)

Outcome measures should be:

  • Valid
  • Reliable
  • Relevant to stage of recovery
  • Sensitive to change
  • Integrated seamlessly within routine clinical practice
slide9

At What Level?

The individual service user

  • Everybody concerned with the care and wellbeing of the person with ABI interested in this level of outcome measurement

ABI Rehabilitation Services

  • Commissioners and purchasers of ABI services (‘efficacy snapshots’)
slide10

‘tactical’

‘strategic’

vs.

Evidence of Homer’s individual rehab outcomes

Evidence of outcomes reflecting the efficacy of Springfield General Hospital as a service

slide11

Lynne Turner-Stokes (1999)

Outcome measures are either:

‘Focal’ vs. ‘Global’

Discuss in the context of individual vs. service level outcome measurement

slide12

‘Focal’ Measures

Distinct behaviours or functions

  • 10 Metre Walk
  • Nin-hole Peg Test
  • Verbal aggression

Often used to measure outcome of specific interventions

  • Can be directly observed and consistently recorded
  • Very flexible, designed to meet individual needs (e.g. Goal Attainment Scaling)
  • Contributes to evidence based knowledge and practice
slide13

Use of methodologies borrowed from

single-case experimental design

can help answer questions asked by

different people by objectively and

validly demonstrating quality outcomes

arising as a consequence of treatment.

For example, for the clinician:

“Is my treatment the cause

of this person’s change?”

(Wilson, 1991)

slide14

This can incorporate principles from single-case experimental design methodology…

baseline

treatment

baseline

treatment

slide15

‘A-B-A-B-C’ Design Showing Reduction in Frequency

of Spitting after ABI

A

baseline

B

DRL

A

baseline

B

DRL

C

DRI

1

5

10

15

20

25

30

35

40

45

slide16

Advantages of Single Case Experimental Design Methodologies at Level of Individual Outcome Measurement

  • person is their own control
  • tailored to meet individual needs
  • flexible
  • few ethical concerns
standardised behaviour rating scales for people with acquired progressive neurological conditions
Standardised Behaviour Rating Scales for People with Acquired & Progressive Neurological Conditions
  • The Overt Aggression Scale - Modified for Neurorehabilitation (OAS-MNR: Alderman, Knight and Morgan, 1997)
  • The St Andrew’s Sexual Behaviour Assessment (SASBA: Knight, Johnson, Alderman, Green, Birkett-Swan & Yorston, in preparation)
slide18

Overt Aggression Scale - Modified for Neurorehabilitation (OAS-MNR)

  • A standardised behaviour rating scale with good psychometric properties suitable for the measurement and assessment of aggression for people with ABI within in- patient settings
  • Each incident of aggression is captured to determine possible relationships between the environment and behaviour (objectively inform formulation)
  • Although evolved primarily for use with ABI, OAS-MNR has seen widespread use with other clinical populations
slide21

B’line

Intervention

slide23

‘Global’ Measures

Provide a holistic view of disability (rating scales)

  • Barthel Index
  • FIM
  • FIM+FAM
  • Self-report measures

Advantages

  • Standardised measures with known psychometric properties (validity, reliability)
  • Broad overview of strengths and weaknesses
  • Highlight targets for rehab
  • Common language providing continuity within and between services
slide24

Example FAM-splat

26-30

Cognitive

Function

1-7

Self Care

8-9

Sphincter

Control

22-25

Psychosocial

Adjustment

10-13

Mobility/

Transfers

17-21

Communication

14-16

Locomotion

Adm

Cur

Goal

slide25

What To Use

Multiple Consequences of ABI:

  • physical
  • functional
  • cognitive
  • emotional
  • psychosocial
  • behavioural

Type of service (generalised, specialised)

slide26

No single outcome measure is suitable for all brain injury rehabilitation

Pick and chose a ‘basket’ of outcome measures

slide27

Most Frequently Used Outcome Measures in UK Rehab Centres (Turner-Stokes & Turner-Stokes, 1997)

82% neurorehab services, 123/180 used global measure (Barthel, FIM, FIM+FAM)

slide28

FIM+FAM

HoNOS

ABI

HoNOS

Secure

slide29

Service Level Outcomes

  • Global measures of disability typically used
  • Data from individual SU’s pooled (average)
  • Provides ‘snapshot’ to commissioners regarding effectiveness of service
slide32

Multiple Consequences of ABI

  • physical
  • functional
  • cognitive
  • emotional
  • psychosocial
  • behavioural
slide33

Particular Difficulties in Using Group Methodologies

to Determine Quality Outcomes with ABI

Non-homogenous group

Mateer & Ruff (1990)

“...no two head injuries are alike”

Wilson (1991)

people present with combination of problems,

rarely identical

  • lack of homogeneity
  • between & within sub-populations
  • needs & strengths
  • different stages of recovery
slide34

Lack of homogeneity

Integrity of measures when data

are pooled

  • group averages obscure individual outcome
  • data distributions can be abnormal
  • variability within data can be extreme
slide35

Effectiveness of a Behavioural Approach in Reducing the

Frequency of Physical Aggression in a Group of 40 Clients

(from Alderman, Bentley & Dawson 1999)

22.9

7.1

slide36

Effectiveness of a Behavioural Approach in Reducing the

Frequency of Physical Aggression in a Group of 40 Clients

(from Alderman, Bentley & Dawson 1999)

72.4

17.3

slide37

Effectiveness of a Behavioural Approach in Reducing the

Frequency of Physical Aggression in a Group of 40 Clients

(from Alderman, Bentley & Dawson 1999)

451

33

slide38

451

82

13

33

Effectiveness of a Behavioural Approach in Reducing the

Frequency of Physical Aggression in a Group of 40 Clients

(from Alderman, Bentley & Dawson 1999)

slide39

Some Ongoing Developments in Measuring Neurobehavioural Outcome

Response to these challenges

  • Generation of a ‘snapshot’ of service level outcomes from an observational behaviour rating scale designed for use with individual SU’s.
  • ‘focal’ measure (aggression)
  • Individual to service level outcome
  • A new rating scale to measure neurobehavioural disability and social handicap.
  • ‘global’ measure
  • Individual and service level outcome
slide40

Creating Service Level Performance Indicators from Routinely Administered Clinical Tools

Can a clinical measure employed to collect data for the purpose of delivering individual care can be utilised as an ongoing indicator of service level outcome?

Advantages of:

  • specificity
  • known reliability/validity
  • savings in clinicians time for training and administration
slide41

OAS-MNR data recorded for 79 service users:

  • ABI acute neurobehavioural wards
  • PNC wards

AAS = total frequency of aggression x mean weekly weighted severity

Aggregate Aggression Score calculated for the first and most recent three months of admission

slide43

Comparison of Median Frequency and AAS Scores

Percentage Median AAS Improvement between First and Last 3 Months of Admission

slide44

Creating Service Level Performance Indicators from Routinely Administered Clinical Tools

  • Pooling data still creates problems with abnormal distributions – use median
  • Incorporating a range of information into a rolling PI gives a more balanced index of change
  • Can be drilled down, e.g. to ward level
  • Crude but valuable ‘snapshot’ to help inform commissioners and other stakeholders re efficacy of service
slide45

Kolitz et al (2003) argued that there was still a need to develop a valid and comprehensive instrument for the measurement of NBD.

slide46

Neurobehavioural Disability

  • executive dysfunction
  • attention deficits
  • diminished insight
  • poor social judgement
  • labile mood
  • problems with impulse control
  • personality change

Wood (2001)

Complex, subtle, pervasive constellation of cognitive-behavioural changes that characterise post-acute ABI

social

handicap

slide48

Kolitz et al (2003) argued that there was still a need to develop a valid and comprehensive instrument for the measurement of NBD.

Wood, Alderman & Williams (2008)

Assessment of neurobehavioural disability:

a review of existing measures and recommendations for a comprehensive assessment tool

Brain Injury, 22, 905–918

  • Undertook review of the psychometric properties of 8 widely used measures of NBD
  • Made proposals regarding development of future measures of NBD
slide49

Wood, Alderman & Williams (2008)

Some of the proposals made regarding development of future measures of NBD included:

  • Measures have clear theoretical underpinning/conceptual framework that drives item selection
  • Information is obtained from an informant
  • Sufficient pool of items to capture the diverse range of NBD signs and symptoms
  • Robust, well known statistical properties
  • (reliability data sparsely reported)
slide50

Wood et al (2008) review did not contradict claim by Kolitz et al (2003) regarding the need to develop a valid and comprehensive instrument for the measurement of NBD

slide51

Two Key Statistical Properties in Outcome Measurement

Reliability

How consistent an outcome tool is.  A measurement is said to be reliable or consistent if the measurement can produce similar results if used again in similar circumstances.

Validity

The extent to which an outcome tool truly measures what it claims to measure

slide52

Validity

  • Content Validity
  • does the scale adequately and representatively sample symptoms characteristic of NBD?
  • Construct Validity
  • does the scale accurately measure the theoretical, non-observable construct that characterise NBD?
  • Criterion-related Validity
  • does the scale accurately predict the presence of NBD/social handicap/ABI?
slide53

Reliability

  • Internal Consistency Reliability
  • are items within the scale that measure the same aspects of NBD rated similarly?
  • Inter-rater Reliability
  • do different people rating the same patient do so similarly?
  • Test-Retest Reliability
  • how similar are ratings made over time?
  • (major implications for recovery and rehabilitation)
st andrew s swansea neurobehavioural outcome scale sasnos
St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Nick Alderman St Andrew’s Healthcare

Rodger Wood Swansea University

Claire Williams Swansea University

  • Designed specifically to capture NBD
  • Goal to create an outcome measurement tool that can be used for clinical work and research
  • Aspire to maximise psychometric properties as recommended in 2008 review of other measures
slide55

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

  • Conceptual framework to develop the new measure
  • was the WHO ICIDH-2, using the levels of Activities
  • and Participation (rather than Pathology and
  • Impairment) – functional difficulties that can be
  • observed within a broader social context
  • Items reflecting NBD and social handicap rated on
  • 7-point Likert scale
slide56

Never

Always

1 2 3 4 5 6 7

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

‘Exhibits sudden/rapid shifts of temperament

and behaviour’

slide57

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

  • Initial pool of 117 items
  • 300 sets of ratings obtained from clinicians of all grades regarding service users admitted to the National Brain Injury Centre
  • Three stages to data collection
  • determine validity, reduce pool of items
  • obtain ratings for neurologically healthy participants from Swansea University
  • determine inter-rater and test-retest validity of final data set
slide58

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Validity

  • 97 service users each rated once on the 117 items by 52 clinicians
  • Some items not rated so these were discarded
  • Exploratory Factor Analysis reduced data set further to 58 items
slide59

Not at all well

Very well

1 2 3 4 5 6 7

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Content Validity– do items capture overt NBD symptoms?

  • Face Validity

After rating a service user on the 117 item data set clinicians were asked to rate the following:

“How well do you feel that the items rated here reflect the sorts of behaviours you encounter in patients with acquired brain injury here at Kemsley?”

slide60

Not at all well

Very well

1 2 3 4 5 6 7

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Content Validity– do items capture overt NBD symptoms?

  • Face Validity
  • Mean rating = 5
  • 70% clinicians rated initial item pool 5 or 6
  • 79% clinicians rated 58-item pool 5 or 6
slide61

Items

% variance

  • Relationships 17 13.5
  • Cognition 15 12.0
  • Aggression 15 11.3
  • Inhibition 6 7.1
  • Communication 5 6.0

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Construct Validity– do items measure underlying constructs?

  • Factorial Validity

Measure of NBD fractionates into 5 separate, but related, factors:

slide62

provocation

Aggression

irritability

overt aggression

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Construct Validity– do items measure underlying constructs?

  • Factorial Validity

Separate analysis of each factor suggests these fractionate into 2-3 further sub factors.

For example:

slide63

Inhibition

sexual

social

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Construct Validity– do items measure underlying constructs?

  • Factorial Validity

Separate analysis of each factor suggests these fractionate into 2-3 further sub factors.

For example:

slide64

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Construct Validity– do items measure underlying constructs?

  • Internal Consistency

Cronbach’s alpha

slide65

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Construct Validity– do items measure underlying constructs?

  • Convergent/Divergent Validity

Overall score, factor scores and sub factor scores correlate well with other measures in the outcome ‘basket’ that measure similar concepts, and poorly with those that measure different domains.

  • HoNOS Secure
  • HoNOS ABI
  • FIM+FAM
  • DEX Questionnaire – Others Version
  • Aggression indices
  • ISB indices
slide66

Convergent/Divergent Validity

FIM+FAM

FREQ.AGG

AGG.SCORE

FREQ.ISB

ISB.SCORE

SASNOS

REL

HoNOS SECURE

COG

HoNOS ABI

AGG

DEX-0

INH

INH

low >.2

mod >.35

COM

high >.5

INT

EXE

+AF

-AF

  • Total Score
slide67

Convergent/Divergent Validity

FIM+FAM

FREQ.AGG

AGG.SCORE

FREQ.ISB

ISB.SCORE

SASNOS

REL

HoNOS SECURE

COG

HoNOS ABI

AGG

DEX-0

INH

INH

low >.2

mod >.35

COM

high >.5

INT

EXE

+AF

-AF

  • Aggression
slide68

Convergent/Divergent Validity

FIM+FAM

FREQ.AGG

AGG.SCORE

FREQ.ISB

ISB.SCORE

SASNOS

REL

HoNOS SECURE

Sexual

Social

COG

HoNOS ABI

AGG

DEX-0

INH

INH

low >.2

mod >.35

COM

high >.5

INT

EXE

+AF

-AF

  • Inhibition Sub factors
slide69

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Criterion-related Validity– predictive value of items?

Determine by comparing ABI and neurologically healthy control group data.

Ratings of close relative/friend made by 100 staff/associates at Swansea University.

Scale should:

  • Discriminate ABI from controls
  • Discriminate controls vs. ABI DES vs. ABI non-DES
slide70

Comparison of ABI and Control SASNOS Ratings

Mean Sum of Ratings

Significant difference between all mean scores at p<.001

Discriminant Function Analysis: 85% cases correctly classified using total SASNOS score

slide71

St Andrew’s-Swansea Neurobehavioural Outcome Scale (SASNOS)

Validity

  • Internal Consistency Reliability
  • Cronbach alpha values satisfactory
  • Inter-rater Reliability
  • 50 patients rated independently by pair of raters – ICC’s should be >.4
  • Test-Retest Reliability
  • clinicians repeated ratings of 41 patients 2- 3 weeks later – these should be similar
slide72

55-58

Communication

Example SASNOS-splat

C2, 57-58

C1, 54-56

R1,1-5

48-53

Inhibition

1-17

Relationships

I2, 51-53

R2

6-12

I1, 48-50

A3, 45-47

R3,13-17

A1, 39-44

33-47

Aggression

C1, 18-25

A1, 33-38

18-32

Cognition

C2, 26-32

Adm

Cur

Goal

slide73

Adm

Cur

Goal

Visual Representation of Functional, Neurobehavioural and Social Outcome Following ABI Rehabilitation

55-58

Communication

26-30

Cognitive

Function

1-7

Self Care

48-53

Inhibition

1-17

Relationships

8-9

Sphincter

Control

22-25

Psychosocial

Adjustment

10-13

Mobility/

Transfers

33-47

Aggression

17-21

Communication

18-32

Cognition

14-16

Locomotion

slide74

Outcome Measurement in ABI

Some final thoughts…

  • No single solution
  • Individual outcome most informative, flexible and greatest choice of methods
  • Outcome ‘basket’ will inevitably differ, but agreement on ‘core’ measures helpful
  • Threats to validity of service level outcomes but ‘snapshots’ have place, advantages to adopting existing measures for that purpose
  • Market for new global NBD measure, SASNOS offered as candidate
slide75

Thanks for listening…

nalderman@standrew.co.uk