slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Gregory P. Beehler, Ph.D., M.A., PowerPoint Presentation
Download Presentation
Gregory P. Beehler, Ph.D., M.A.,

Loading in 2 Seconds...

play fullscreen
1 / 44

Gregory P. Beehler, Ph.D., M.A., - PowerPoint PPT Presentation

  • Uploaded on

Session #E5a Saturday, October 12, 2013. Anything Goes? Developing a New Measure of Fidelity for Behavioral Health Providers in Integrated Primary Care. Gregory P. Beehler, Ph.D., M.A., Clinical Research Psychologist, VA Center for Integrated Healthcare Jennifer S. Funderburk, Ph.D.,

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Gregory P. Beehler, Ph.D., M.A.,' - decker

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Session #E5a

Saturday, October 12, 2013

Anything Goes? Developing a New Measure of Fidelity for Behavioral Health Providers in Integrated Primary Care

Gregory P. Beehler, Ph.D., M.A.,

Clinical Research Psychologist, VA Center for Integrated Healthcare

Jennifer S. Funderburk, Ph.D.,

Clinical Research Psychologist, VA Center for Integrated Healthcare

Katherine M. Dollar, Ph.D.,

Technical Assistance Specialist, VA Office of Mental Health Operations

Collaborative Family Healthcare Association 15th Annual Conference

October 10-12, 2013 Broomfield, Colorado U.S.A.

faculty disclosure
Faculty Disclosure

We have not had any relevant financial relationships during the past 12 months.

  • Define fidelity of implementation as applied to behavioral health providers in Integrated Primary Care
  • Identify potential sources of low and high fidelity among behavioral health provider behaviors in primary care
  • Describe the psychometric properties and intended use of a new measure of fidelity for behavioral health provider behaviors in primary care
  • Describe VA behavioral health provider reactions to a field test of a new measure of fidelity
presentation overview
Presentation Overview
  • Background: IPC and fidelity
  • Study 1: Developing a measure of protocol adherence
  • Study 2: Psychometric validation of the PPAQ
  • QI Project: The PPAQ Minikit
  • Implications and Future Directions
  • The role of Behavioral Health Providers (BHPs) in Integrated Primary Care (IPC) is complex
    • The competencies of BHPs are broad1,2
    • BHPs may have limited background and training related to integrated care
    • BHPs face a number of practice barriers that can lead to variability in how IPC is enacted locally3
  • Despite the growing popularity of IPC, the evidence base remains limited, but growing4-6

1. Pomerantz et al, 2009; 2. APA Workgroup, 2013; 3. Beehler & Wray, 2012; 4. Cigrang, et al., 2009; 5. Bryan, et al, 2009; 6. Ray-Sannerud, et al, 2012.

assessing provider behavior
Assessing Provider Behavior
  • How well is IPC being implemented by BHPs? How do we describe, measure, and monitor their behavior?
  • Fidelity is the most commonly reported measure of implementation outcomes1
    • Fidelity refers to the degree to which an intervention is delivered as intended
      • Also referred to as treatment integrity or quality of implementation
    • It is a construct that appears in many evaluation frameworks, such as RE-AIM2

1. Proctor, et al., 2011 ; 2. Glasgow, et al., 1999

measuring provider fidelity
Measuring Provider Fidelity
  • Fidelity can be assessed at several levels
    • molar(program-level)
    • molecular (provider-level, or protocol adherence)
  • Protocol adherence isthe degree to which a provider utilizes certain procedures and engages in specific tasks and activities1
    • Are patients receiving the intervention’s active ingredients from the provider?
    • The “bottom line” of implementation fidelity2

1. Perepletchikova & Kazdin, 2005; 2. Carroll et al, 2007.

conceptual model fidelity of implementation
Conceptual Model: Fidelity of Implementation

Patient, Provider, Clinic, and System Barriers and Facilitators

Protocol Adherence

Patient Outcomes: Distress, Coping, Adherence, etc.

Integrated Primary Care

  • Carroll C, Patterson M, Wood S, Booth B, Rick J, Balain S. A conceptual framework for implementation fidelity. Implementation Science. 2007; 2: 40.

Potential Moderators:

.Comprehensiveness of policy

.Strategies to facilitate implementation

.Quality of Delivery

.Participant Responsiveness


Details of content






Evaluation of Implementation Fidelity


Component analysis to identify “essential” components

why invest in fidelity assessment
Why Invest in Fidelity Assessment?
  • Develop IPC measures:
    • Currently, there are very few validated measures of provider behaviors
  • Advance IPC research:
    • Linking provider behaviors to patient outcomes
      • What formulation of IPC do we use?
  • Training and Implementation:
    • Can a protocol adherence measure assist BHPs in understanding what specific behaviors constitute high quality practices?
delphi study approach
Delphi Study Approach
  • Goal: Build consensus on critical BHP behaviors that should be included in a measure of protocol adherence
  • Design: Modified Delphi study
    • Structured, iterative process of collecting and summarizing opinions from content experts
    • Often used to address problems of clinical practice when there is incomplete knowledge
    • Improves content validity for instrument development

Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. (2013). Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: A Delphi study. Implementation Science, 8, 19.

delphi study approach1
Delphi Study Approach
  • A preliminary set of 56 items were developed by the research team over six months
    • Items represented critical provider behaviorsrelated to clinical services or collaboration
    • Items based on a qualitative review of empirical and theoretical works
  • Each item was classified by the team based on relevance for IPC:
    • Essential(consistent with the IPC model; required for good practice)
    • Prohibited(inconsistent with the IPC model; should be avoided)
    • Compatible (neither essential nor prohibited)
delphi study participants
Delphi Study Participants
  • Delphi panel sizes vary considerably
    • We aimed for complete participation from 20 experts
  • 33 IPC experts from VA, Department of Defense, and academic health centers were invited to participate
    • Purposefully sampled based on clinical, administrative, or research experience
  • 25 experts (76% response rate) participated in three rounds of emailed surveys
selected participant characteristics
Selected Participant Characteristics

*n=24 due to missing data.

delphi process results
Delphi Process Results
  • Participants provided qualitative feedback on all items and also suggested new items
  • Participants were asked to classify each item as essential, prohibited, or compatible
    • Items that reached ≥80% consensus were considered complete
    • 19 items met consensus in Round 1
    • 26 additional items met consensus in Round 2
    • 9 additional items met consensus in Round 3
  • Final results at the end of Round 3:
    • 38 essential, 10 prohibited, and 6 compatible behaviors
ppaq item examples
PPAQ Item Examples
  • The resulting 54-item instrument was named the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ)
  • Item 11 (essential)

My progress notes in the shared medical record include focused recommendations for the Primary Care Provider and/or primary care team

  • Item 44 (prohibited)

I typically see patients for 10 or more appointments per episode of care

  • Item 31 (compatible)

I conduct follow-up appointments via telephone when appropriate

  • A modified Delphi study was used to reach consensus on 54 critical provider behaviors related to the delivery of IPC
  • 93% of original items developed by the research team were included in the final instrument representing several IPC domains:
    • Clinical scope and interventions
    • Practice and session management
    • Referral management and care continuity
    • Consultation, collaboration, and interprofessional communication
  • The expert panel ensured content validity, but reliability and construct validity required testing with frontline providers

Study 2: Psychometric validation of the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ)

ppaq study objectives
PPAQ Study Objectives

Goal: To evaluate the psychometric properties of the PPAQ using a sample of behavioral health providers (BHPs) working in VA primary care settings


  • BHPs working solely in the CCC (co-located, collaborative care) role would have higher fidelity compared to those working in either care management-only, or CCC and care management combined roles
  • BHPs working in CCC settings at VA Medical Centers would evidence higher fidelity scores than BHPs working in CCC settings at VA Community-Based Outpatient Clinics
ppaq study objectives1
PPAQ Study Objectives

Hypotheses (continued)

  • BHPs with a longer history of providing CCC services would have higher fidelity ratings due to mastery over CCC-specific roles compared to those BHPs with a shorter history of CCC work
  • BHPs who self-reported higher levels of knowledge/skills related to CCC would show higher levels of fidelity
  • BHPs who self-reported greater access to material and resources to deliver CCC would show greater fidelity
  • BHPs who endorse a CBT orientation over other forms of psychotherapy would demonstrate higher fidelity
  • Look at PPAQ scores with scores from the Level of Integration Measure (LIM) (a self-report measure)
ppaq study design
PPAQ Study Design
  • Email invitation to online survey consisting of consent information, background information, and questionnaires
    • BHP Background Questionnaire
      • Relevant professional background characteristics
      • Knowledge and access to resources consistent with CCC
    • PPAQ
      • 54-item, 5-point Likert-like scale of BHP protocol adherence to the CCC model
      • 3 subscales: PPAQ-E (38 essential items), PPAQ-C (6 compatible items), PPAQ-P (10 prohibited items)
    • LIM
      • 35-item, 5-point Likert-like scale reflecting level of behavioral and physical health integration in primary care
      • 6 subscales addressing aspects of integration

Beehler, G. P., Funderburk, J. S., Possemato, K., & Dollar, K.M. (2013). Psychometric assessment of the Primary Care Behavioral

Health Provider Adherence Questionnaire (PPAQ). Translational Behavioral Medicine, 1, 13.

ppaq study participants
PPAQ Study Participants
  • Potential participants identified by previous participation in PC-MHI-related research or who attended PC-MHI educational trainings from 2009-2011
  • 580 BHPs who provided clinical services in primary care for at least 25 % of their duties, had an active VA email account, and with sufficient time to complete a brief (i.e., <20 min) online survey were invited to participate
  • 173 BHPs provided complete responses to the survey, yielding a 30 % response rate (on par with other electronic survey studies)
participant characteristics continued
Participant Characteristics continued
  • Majority of BHPs endorsed a CBT orientation (60%)
  • Among those endorsing a CCC role:
    • 26% felt they lacked resources to provide CCC
    • 94% agreed that they had the knowledge and skills to provide CCC
    • Encouraging, but due to limited variability, this variable was not included in subsequent analyses
ppaq results
PPAQ Results
  • Reliability
    • PPAQ-E (alpha=0.92) and PPAQ-P (alpha=0.70) had strong internal consistency
  • Known-groups validity
    • Among BHPs in CCC-only or combined roles, those who had sufficient resources to enact CCC had higher PPAQ-E scores
    • Greater adherence to CCC essential behaviors was associated with CBT orientation and greater perceived resources for CCC
  • Convergent and discriminant validity
    • Highly significant low to moderate correlations of PPAQ-E and LIM total score (r=0.52, p<0.001)and subscales (ranging from 0.35 to 0.49, p<0.001) suggest good convergent validity
    • Non-significant correlations between PPAQ-P and LIM total score and subscales suggest good discriminant validity
    • Lack of correlation between PPAQ-E and PPAQ-P (r=0.10, p=.197) suggest good discriminant validity
  • Relied on self-report for the nature of the professional role (CCC, care manager, or both)
  • Based sampling on previous participation in CCC-related trainings or research, thus the participants may report increased knowledge or familiarity with CCC practice
    • May also have social desirability bias
  • Validation in non-VA samples is needed
  • Methodology did not allow investigation of multiple forms of construct validity
  • No well-accepted “gold standard” measures of CCC adherence with which to compare
  • The PPAQ has emerged as the first reliable self-report instrument of BHP fidelity that has demonstrated strong evidence of criterion validity
  • Implications: beneficial for provider training (self-assessment), program monitoring, and clinical research
  • PPAQ has the potential to identify service delivery gaps, but still important to maintain a stance in which fidelity assessment is used to support BHPs’ efforts to improve their capacity to deliver evidence-based, patient-centered care
from measure to minikit
From Measure to Minikit
  • The PPAQ is a psychometrically-sound self-report measure that allows BHPs to assess their usual clinical practices in terms of protocol adherence
    • But what would make the PPAQ more useable for frontline providers?
  • Goal: Develop a fidelity-focused toolkit for CCC providers based on the PPAQ
    • Aim 1: Prepare the PPAQ Minikit components
    • Aim 2: Pilot the PPAQ Minikit with CCC providers to evaluate acceptability and utility of the PPAQ Minikit as a diagnostic assessment that directs future implementation support

Beehler, G. P., & Dollar, K.M. (2013). Employing an Evidence-Based Fidelity Tool to Identify Implementation Support Needs and Preferences among Co-located, Collaborative Care Providers. MH QUERI QLP 55-020.

aim 1 develop ppaq minikit components
Aim 1: Develop PPAQ Minikit Components
  • Work completed:
    • Audio-enabled presentation providing background on the PPAQ and instructions for using the Minikit
    • PPAQ self-report form
    • Scoring template and interpretation guide
moving forward a process for change
Moving Forward: A Process for Change
  • 1. Prioritize areas for change
  • Identify areas for improvement based on PPAQ item analysis
  • Rank order areas by feasibility
  • Start small: Select one or two behaviors to address

Complete Scoring Template and Interpretation Guide

  • 2. Partner with advocates and peers
  • Reach out to PC-MHI Champion, colleague, or supervisor
  • Join local/regional peer support networks
  • Request consultation from CIH or PC-MHI PO

Initial PPAQ self-administration

Re-administer PPAQ

  • 3. Access educational and clinical resources
  • Review PC-MHI PO resources
  • Review CIH practice tools and educational supports
  • Consider PC-MHI relevant handbooks and journals
  • 4. QualityImprovement
  • Specify attainable practice change goals
  • Conduct P-D-S-A cycles to reach individual goals
  • Use a team-based approach for larger clinical goals
aim 2 piloting
Aim 2: Piloting
  • Methods
    • Convenience sample of 10 VA BHPs
    • Pilot the PPAQ Minikit components available via SharePoint
    • Complete a telephone interview to identify key strengths and weaknesses related to acceptability, quality, and likelihood of practice change
      • Calculate estimates of acceptability
      • Identify themes related to knowledge and practice change
preliminary results
Preliminary Results
  • Strengths:
    • Overall usability and acceptability of the Minikit as a diagnostic self-assessment was rated highly
    • Minikit item content was perceived as very reflective of the IPC model
    • The tone of the Minikit was described as non-punitive and balanced because it identified areas of strength as well as weaknesses
    • Participants noted that the Minikit would be highly useful for BHPs new to the IPC/CCC role to clarify “do’s and don’ts”
      • Mature BHPs also found it to be a helpful reminder to avoid drift
preliminary results1
Preliminary Results
  • Weaknesses:
    • Limited familiarity with Excel was a barrier to usability of Scoring Template and Interpretation Guide
    • Current form of the Minikit may be seen as a one-time assessment rather than a tool to direct on-going learning
    • Improving adherence remains challenging in some contexts due to clinic and system barriers (e.g., limited time for QI projects or self-study or unwillingness of primary care providers to fully collaborate)
summary and implications
Summary and Implications
  • The PPAQ is the first reliable, validated measure of BHP protocol adherence
  • The PPAQ can be used in future effectiveness research to link provider behaviors to patient outcomes
  • The PPAQ may be used alone or with other program development activities (e.g., facilitation and coaching) to monitor provider behavior
  • The PPAQ Minikit has shown early good potential to be a provider-focused, self-assessment and quality improvement package
audience q a disrupting your status quo
Audience Q&A:Disrupting Your Status Quo?
  • Is your status quo “anything goes”?

How do you rate on these PPAQ sample items using the following scale?

1=Never, 2=Rarely, 3=Sometimes, 4=Often, and 5=Always

1. During clinical encounters with patients, I see patients for 30 minutes or less.

26. I provide education to the primary care team on behavioral health issues (e.g., presentations and handouts).

28. I provide family or couples therapy for 10 or more appointments per episode of care.


How do you rate on these PPAQ sample items using the following scale?

1=Never, 2=Rarely, 3=Sometimes, 4=Often, and 5=Always

32. During a patient appointment, I provide full length empirically supported treatments, such as Prolonged Exposure or Dialectical Behavior Therapy.

43. I meet with a patient for greater than 50 minutes to gather a full psycho-social history and comprehensive psychiatric interview.

48. I participate in primary care based clinical pathways for common health conditions, such as chronic pain or comorbid depression and cardiovascular disease.

  • PPAQ Team
  • Kyle Possemato, Ph.D.
  • Christina Vair, Ph.D.
  • Margaret Dundon, Ph.D.
  • Support
  • VA Center for Integrated Healthcare Pilot Grant Program
  • VA Mental Health Quality Enhancement Research Initiative (QLP 55-020)
session evaluation
Session Evaluation

Please complete and return theevaluation form to the classroom monitor before leaving this session.

Thank you!