slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior PowerPoint Presentation
Download Presentation
Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior

Loading in 2 Seconds...

play fullscreen
1 / 30

Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior - PowerPoint PPT Presentation

  • Uploaded on

Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior. Philip Hemphill, Ph.D. Professional Enhancement Program (PEP) Pine Grove Behavioral Health and Addictive Services FSPHP Presenter Disclosure Slide.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior' - hadar

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior

Philip Hemphill, Ph.D.

Professional Enhancement Program (PEP)

Pine Grove Behavioral Health and Addictive Services


FSPHP Presenter Disclosure Slide

“Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior”

Philip Hemphill, PhD

is a paid consultant of

Pine Grove Behavioral Health and Addiction Services


Annual Conference and Meeting

Fort Worth, Texas

April 23-26, 2012

why use a msf 360
Why use a MSF/360?
  • Facilitate insight of strengths and weaknesses for an individual.
  • Enhance organizational cultural change.
  • Provide summative assessment of performance.
  • Evaluate potential.
  • Improve team effectiveness.
  • Identify training & coaching needs.
  • Make organizational values explicit.
  • Measure progress after training & coaching.
msf 360 essentials
MSF/360 Essentials
  • Proven validity & reliability.
  • Useful in assessing “humanistic” skills.
  • Weak or no associations between self-rated assessment and external assessment.

“…methods such as multisource feedback (360) evaluations may be

a necessary next step, particularly when interpersonal, communication

skills, or professionalism needs to be evaluated (p. 1101).”

Source: Davis et al. (2006). Accuracy of physician self-assessment compared with observed

Measures of competence.JAMA, 296 (9), 1094-1102.

key players in msf 360
Key Players in MSF/360







msf 360 internal process
MSF/360 Internal Process


Is it consistent with self-perception?

Emotional Response

Reflection and internal processing of

emotion and content.

Decision to accept and act, or not

Action for learning, change

Source: Sargeant et al. (2009)

Advances in Health Sc Ed.

Relevant Literature Review of Multi-Source Feedback MeasuresThe Fundamental Interpersonal Relations Orientation–Behavior™ (FIRO-B®)
  • In the late 1950s William Schutz, PhD. developed the FIRO-B theory to aid in the understanding and predicting of how high-performance military teams would work together.
  • In developing the FIRO-B theory, Schutz began with the premise that "people need people." He used the term interpersonal to indicate any interaction, real or imagined, occurring between people. He used the term need to describe a psychological condition that, if not satisfied, leads to a state of discomfort or anxiety.
  • He posited that interpersonal needs could be grouped into three categories: Inclusion, Control, and Affection. The FIRO-B model describes the interaction of these three categories of interpersonal need along two dimensions: expressed and wanted.
  • Awareness Scales and were designed primarily to help individuals with their self-awareness and of understanding their relation to other people (Schutz, 1978).
relevant literature review of multi source feedback measures
Relevant Literature Review of Multi-Source Feedback Measures

Developing insight into strengths and weaknesses, for the individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio & Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999; McCarthy & Garavan, 1999) and the organization (London & Beatty, 1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow & London,1998) to act upon.

Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994; Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998; DeSimone, 1998).

Summative assessment of performance (Newble et al., 1999).

Evaluating the potential of individuals, for instance to use in career advice or selection (Tornow, 1993; Lindsey et al.,1997; Facteau et al., 1998).

Enhancing team effectiveness (by allowing opportunity to comment) (Garavan et al., 1997; Towers-Perrin, 1998).

Identifying training needs for the system (as opposed to the individual) (Towers-Perrin, 1998).

development of disruptive behavior instrument
Development of Disruptive Behavior Instrument
  • 2004 – Developed list of disruptive behaviors and boundary violation behaviors. Approximately 70 items…
  • 5 people monitored over 12 months – Instrument too long; complaints from participants; descriptive data….
  • 2005- Cut instrument down to 50 items(1 Fctr)
  • 2008- Cut down to current version
  • Been using current version for past 4 years…
definition of disruptive behavior
Disruptive behavior is any inappropriate behavior, confrontation, or conflict, ranging from verbal abuse to physical or sexual harassment. Disruptive behavior causes strong psychological and emotional feelings, which can adversely affect patient care. Definition of Disruptive Behavior

Rosenstein A, O’Daniel M. (2008). Managing disruptive physician behavior: Impact on staff relationships. Neurology, 70, 1564-1570.

disruptive behavior items
Avoids taking responsibility for actions

Inappropriately blames others

Places other interests above the safety and welfare of community members

Suspects, without sufficient basis, that others are exploiting, harming, deceiving, or out to “get” him

Seems to have a preoccupation with unjustified doubts about the loyalty or trustworthiness of others

Reads hidden, demeaning, or threatening meanings into benign remarks or events

Perceives attacks on his character/reputation not apparent to others and is quick to react angrily/counterattack

Shows rigidity and stubbornness

Displays an unpredictable mood

Responds to constructive criticism in an immature/irresponsible fashion

Takes too long to requests from others

Arrives late

Fails to complete paperwork in a timely manner

Responds to requests in an avoidant, unreliable, and/or uncooperative manner

Displays a sense of entitlement

Shows perfectionism that interferes with task completion Behaves in a sexually inappropriate manner (verbally or physically)

Lacks empathy Intimidates others or attempts to intimidate others

Exhibits physically aggressive or assaultive behaviors

Exhibits arrogant behaviors and/or has an arrogant attitude

Demonstrates an inflated sense of self-importance

Takes advantage of others in order to get what he wants

Is reluctant to delegate tasks or to work with others unless they submit to exactly his way of doing things

Expresses anger/hostility inappropriately

Exhibits passive-aggressive behavior

Uses abusive, belittling, condescending, demeaning, and/or threatening language

Disruptive Behavior Items


Client signs release allowing for electronic correspondence, and client given monitoring instructions and materials with due date.

Client identifies MC, gives MC instructions and rating materials, and with MC identifies approx. 15 raters..

Client completes treatment

Client participates in interventions

while continuing rating process

Rating process is repeated at predetermined intervals until four uneventful reports

Ratee email addresses



Client attends treatment and rating process is temporarily suspended

PHP/LAP recommends interventions

No significant impairment is present

Collection of surveys online.


recommends intensive/

residential treatment

Significant impairment is present and PHP/LAP intervenes

Client, MC, and other relevant stakeholders review the feedback report

Report produced and emailed to persons indicated on release.


Online Survey

33 questions

  • 28 Likert-type questions regarding ratee’s behavior
  • 1 question concerning how many hours rater works with ratee
  • 1 question about the rater’s title
  • 3 open-ended questions that solicit comments
survey dimensions
Appropriate Documentation





Exploitation of Others

Expression of Anger


Job Attitudes

Multicultural Sensitivity

Openness to Others


Respect for Others

Response to Criticism

Responsibility for Actions


Satisfaction with Ratee

Sexual Appropriateness

Stability of Mood

Stress Management




Team Impact

Threatening Behavior

Verbal Aggression

Working with Others

Survey Dimensions
disruptive behavior instrument
Disruptive Behavior Instrument
  • Descriptive Characteristics of Sample:
    • N = 22 ratees (498 surveys)
      • Average Age = 38 years
      • Male = 19 ; Female = 3
    • Survey Breakdown by rater:
      • Self-Report = 53
      • Staff = 328
      • Supervisor = 37
      • Peer = 79
    • Number of Times each patient was assessed:
      • Range – 1-8 times
disruptive behavior instrument1
Disruptive Behavior Instrument
  • Originally hypothesized to be based
  • on 5 factors
  • Sexually inappropriate behavior
  • Sexual Inappropriateness
  • Sexual/Inappropriate Remarks
  • Aggression
  • Aggression: Non-verbal
  • Aggression: Physical
  • Aggression: Verbal
  • Aggression: Written
  • Self Regulation
  • Mood Instability
  • Poor Stress Management
  • Interpersonal Difficulties
  • Intimidation
  • Lack of Empathy
  • Lack of Multicultural Sensitivity
  • Lack of Openness to Others
  • Problematic Perfectionism
  • Response to Criticism
  • Rigidity
  • Suspiciousness
  • Disruptive Behavior
  • Arrogance
  • Blaming Others
  • Disobeying Rules
  • Disrespect for Staff/Colleagues
  • Ease of Working with Ratee
  • Employee Turnover Intentions
  • Exploitation of Others
  • Poor Citizenship
  • Tardiness
disruptive behavior instrument2
Disruptive Behavior Instrument

Factor 1

Factor 2

Factor 3

  • Results of hypothesized 5-factor loadings; 59% cumulative total variance explained

Factor 4

Factor 5

raw data eigenvalues mean percentile random data eigenvalues
Raw Data Eigenvalues, Mean & Percentile Random Data Eigenvalues

Ncases 498; Nvars 27; ; Ndatsets 1000; Percent 95

Root Raw Data Means Percentile

1.000000 10.462501 1.456515 1.528056

2.000000 1.836325 1.386788 1.437043

3.000000 1.408055 1.335702 1.380089

4.000000 1.231678 1.291131 1.331221

5.000000 1.178393 1.253103 1.287905

6.000000 .966591 1.216903 1.247532

7.000000 .890236 1.183359 1.212107

8.000000 .805858 1.152611 1.180525

9.000000 .786433 1.121760 1.146968

10.000000 .707731 1.092700 1.117568

11.000000 .671235 1.064358 1.087904

12.000000 .647008 1.037406 1.061681

13.000000 .567647 1.010437 1.033661

14.000000 .518775 .984026 1.007120

15.000000 .514043 .958262 .981678

16.000000 .454498 .932558 .955167

17.000000 .414122 .907291 .929837

18.000000 .395714 .882943 .906811

19.000000 .363423 .857912 .882594

20.000000 .337990 .832012 .855692

21.000000 .312618 .806413 .831047

22.000000 .294869 .780470 .806496

23.000000 .281812 .753717 .780180

24.000000 .268155 .725304 .753368

25.000000 .258453 .695053 .724017

26.000000 .230247 .661538 .694250

27.000000 .195588 .619728 .658277

disruptive behavior instrument3
Disruptive Behavior Instrument

Factor 1

Factor 2

Factor 3

  • Recent analyses reveal that the items are loaded on three distinct factors

* Excluded Item 26

disruptive behavior instrument4
Disruptive Behavior Instrument

Total Variance Explained

disruptive behavior instrument5
Disruptive Behavior Instrument

Component Correlation Matrix

disruptive behavior instrument6
Disruptive Behavior Instrument

Reliability for Three Factors

  • Poor reliability for items on Factor 3 suggests that a 2 Factor structure may
  • be a better fit.
disruptive behavior instrument 2 factors
Disruptive Behavior Instrument2 Factors
  • Interpersonal Aggression
  • Aggression: Non-verbal
  • Aggression: Physical
  • Aggression: Verbal
  • Aggression: Written
  • Arrogance
  • Blaming Others
  • Disobeying Rules
  • Exploitation of Others
  • Intimidation
  • Lack of Multicultural Sensitivity
  • Poor Citizenship
  • Suspiciousness
  • Professionalism
  • Disrespect for Staff/Colleagues
  • Lack of Empathy
  • Lack of Openness to Others
  • Mood Instability
  • Poor Stress Management
  • Problematic Perfectionism
  • Response to Criticism
  • Rigidity
  • Tardiness
disruptive behavior instrument7
Disruptive Behavior Instrument
  • Instead of a third factor, items were named “critical items”.
  • 5 Critical Items:
    • “I think about quitting or transferring to a different job because of

the ratee.”

    • “Touches or looks at one or more patients/coworkers in an

inappropriate/sexually suggestive manner.”

    • “Makes sexual remarks in the workplace that are inappropriate

or tells jokes that makes others uncomfortable.”

    • “Performs unnecessary exams on female and/or male patients.”
    • “Demonstrates lax boundaries when prescribing controlled

substances, giving out samples, or storing samples of

controlled substances.”

disruptive behavior instrument8
Disruptive Behavior Instrument
  • Conclusions/Limitations:
    • We have 2 main factors and 5 critical items.
    • Two factors:
      • Interpersonal Aggression
      • Professionalism
future directions
Future Directions:
  • Measuring and refining Inter-rater Reliability
  • Measuring and refining Test-Retest Reliability
  • How is this data utilized?
  • What are cutoff scores?
  • Can we track disruptive behavior without treatment?
  • Can instrument be utilized outside of medicine?