beyond the checklist assessing understanding of hiv vaccine trials in south africa n.
Skip this Video
Loading SlideShow in 5 Seconds..
Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa PowerPoint Presentation
Download Presentation
Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa

Loading in 2 Seconds...

  share
play fullscreen
1 / 16
Download Presentation

Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa - PowerPoint PPT Presentation

brasen
116 Views
Download Presentation

Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa Graham Lindegger Xolani Xaba Cecilia Milford Catherine Slack Michael Quayle Eftyhia Vardas HIV AIDS Vaccines Ethics Group (HAVEG) www.saavi.org.za/haveg.htm XVI International AIDS Conference 13-18 August 2006

  2. Outline • Background • Study aim, design and results • Implications XVI International AIDS Conference 13-18 August 2006

  3. Background: Complexities • What should be understood? What level? How? How best to measure? • Limitation of traditional methods: • Focus on memory of technical information • May be less culturally congruent • Why alternate methods? • More personal understanding; less technical focus • More culturally sensitive XVI International AIDS Conference 13-18 August 2006

  4. Aims and process • Primary objective: • To compare four methods of assessing understanding of trial participation (i.e., self-report, checklist; story; case) • A collaborative process was followed: • Series of 8 meetings to develop protocol and obtain site buy-in • Ethical approval; Sponsor negotiation; Pilot XVI International AIDS Conference 13-18 August 2006

  5. What concepts did we test? • Collaborative meetings with stakeholders: • What must TPS understand? Prioritize? • Key components: • Trial aims • Eligibility to participate • Methodology (randomisation, placebo, blinding) • Risk of false sense of security • Risk of false positivity • Compensation for research related injury • Informed consent (right to withdraw) XVI International AIDS Conference 13-18 August 2006

  6. What measures did we develop? • Self reported understanding: How do you rate your understanding of [concept]? [no/little, good enough] • Formal checklist with true-false items for each concept • Narrative (story) describing trial participation • Responses to hypothetical vignettes (case examples) of eachconcept (or part thereof) XVI International AIDS Conference 13-18 August 2006

  7. How did we develop a scoring system? • Developed collaboratively during stakeholder meetings • Asked the question: • What would satisfy you that a participant understood enough of this component to be enrolled in the trial? • Discussion – disagreement – resolved by discussion – consensus • Ended up with detailed operational criteria for scoring responses • Allowed development of a “Gold standard” of understanding for each component XVI International AIDS Conference 13-18 August 2006

  8. Using the scoring system • Two point scale with little/no or good enough understanding used • Self-report: Participants estimated own understanding • Checklist: Correct scores on certain items • Narrative: description of essential information • Vignette: describe part of concept XVI International AIDS Conference 13-18 August 2006

  9. Sample and procedure • Sample: • 53 Vaccine Discussion Group/Information Session Members • 26 male, 33 female • 18-55 years • Procedure: • All assessment methods administered in 1 sitting • Self report first, then varied order XVI International AIDS Conference 13-18 August 2006

  10. Data analysis • 2 independent scorers • Understanding of each trial concept across the different measures compared using Cochran’s Q • Overall understanding of trial participation across measures compared using Friedman’s (expressed as %) • Level of agreement between different components on each measure (Cohen’s Kappa) XVI International AIDS Conference 13-18 August 2006

  11. Results • Significant differences across measures of scores of overall understanding • Overall scores were ranked as follows: • Self-report: 86% • Checklist: 82% • Narrative: 69% • Vignette: 67% • Significantly different proportions of ‘good enough’ scores across all measures XVI International AIDS Conference 13-18 August 2006

  12. Component Component Percentage of participants rated as “Good Enough” per section Percentage of participants rated as “Good Enough” per section Cochran’s Q or Friedman’s Cochran’s Q or Friedman’s Asymp Sig. Asymp Sig. Self Report Self Report Checklist Checklist Vignette Vignette Narrative Narrative Trial aims Trial aims 95% 95% 93% 93% 53% 53% 60% 60% Q = 45.192 Q = 45.192 0000* 0000* Eligibility to participate Eligibility to participate 96% 96% 86% 86% 84% 84% 95% 95% Q = 7.929 Q = 7.929 0048** 0048** Methodological considerations Methodological considerations 72% 72% 83% 83% 67% 67% 67% 67% Q = 7.714 Q = 7.714 0052 0052 Risk (False sense of security) Risk (False sense of security) 88% 88% 67% 67% 72% 72% 76% 76% Q = 8.509 Q = 8.509 0037** 0037** Risk (False positivity) Risk (False positivity) 87% 87% 75% 75% 55% 55% 60% 60% Q = 23.76 Q = 23.76 0000* 0000* Compensation for research related injury Compensation for research related injury 70% 70% 85% 85% 44% 44% 43% 43% Q = 32.353 Q = 32.353 0.000* 0.000* Right to withdraw Right to withdraw 96% 96% 95% 95% 93% 93% 98% 98% Q = 3.333 Q = 3.333 0343 0343 Overall/Total understanding (%) Overall/Total understanding (%) 86% 86% 82% 82% 67% 67% 69% 69% 0.000** 0.000** *significant at  = 0.01; **significant at  = 0.05 *significant at  = 0.01; **significant at  = 0.05 Frequency of “good enough” understanding for each subscale XVI International AIDS Conference 13-18 August 2006

  13. Results: Closed versus open XVI International AIDS Conference 13-18 August 2006

  14. Discussion • Measured level of understanding is a function of the assessment method used • Forced-choice checklists and self-reports may be over-estimating levels of understanding, as measured against “gold standard” developed by staff • Self report – social desirability? • Checklist - acontextual rote memorizing of technical information? • Vignettes and narratives - contextually embedded task? • Possibly closer to actual decision-making for trials XVI International AIDS Conference 13-18 August 2006

  15. Implications • Limits of open ended methods: • more time consuming • require greater training to implement • For critical concepts where consequences of misunderstanding are great – supplement with open-ended measures XVI International AIDS Conference 13-18 August 2006

  16. Acknowledgements and thanks • Trial site staff • Study participants • Sponsors THANK YOU XVI International AIDS Conference 13-18 August 2006