Lessons learned from implementation of a mixed method Assessment of Understanding (AOU) tool

1. Lessons learned from implementation of a mixed method Assessment of Understanding (AOU) tool • J.Mbogua1; G. Lindegger2; S.Welsh3; J.Lehrman3, S.Singh3, G. Mutua4, G. Oino4, D.Nyasani4, D. Kerubo4, J. Ng'ang'a4, R. Sajabi4, R. Murungi 4, J. Mukamuyango5, J.Nyombayire5, M. Mukankuku5, R. Bayingana5, J. Mukamwezi5, R. Ingabire5, C.Konde6, J. Mpendo6, A. Nanvubya6, M. Balyeku6, S.Coutinho6 • Institutional Affiliation: • International AIDS Vaccine Initiative (IAVI), Nairobi, Kenya; • HIV/AIDS Vaccine Ethics Group (HAVEG); • IAVI, New York, USA; • Kenya AIDS Vaccine Initiative (KAVI); • Projet San Francisco - Kigali; • Uganda Virus Research Institute (UVRI)-IAVI

2. Background As stipulated in Good Clinical Practice (GCP) guidelines, it is an ethical requirement for volunteers’ to provide informed consent to participate in any clinical trial. Volunteer understanding of trial concepts is an essential component of informed consent. They must understand: • Trial methodology • Product-related information • Personal consequences of trial participation Various tools to determine volunteers’ understanding of concepts have been developed and implemented over time.

3. Rationale Standard Assessment of Understanding (AOU) tools have generally used closed-ended true/false questionnaires. However, literature and anecdotal evidence reveal that closed-ended tools: • May overestimate understanding • Measure short term memory, not understanding • May be culturally foreign Open ended tools, such as scenarios, may be preferable. A tool consisting of both closed ended and open ended questions was developed and tested by IAVI, HAVEG and partners

4. Background of the AOU Tool Development Development of the tool was informed and tested in a series of studies • Lindegger et al (2006), UKZN HAVEG Study • AOU Pilot study • Comparative study

5. Testing multiple tools Lindegger et al examined how well different tools assessed understanding of informed consent concepts: • True/false questionnaires • Self reports • Narratives • Scenarios

6. HAVEG study continued Scenarios Example: Mr. X decides that he will enroll in this HIV vaccine trial. So, he starts attending information sessions/discussion groups. On the way to attend one of his sessions at the vaccine trial site, he thinks to himself, “Once I am in the vaccine trial, I won’t need to worry about practicing safe sex anymore”. If Mr. X told you this what would you say to him? Concept: False sense of security Standardized prompts used to guide conversation.

7. HAVEG study continued Findings • Closed methods (T/F) produced higher scores of understanding –> overestimate understanding? • Concepts differ in complexity – serious impact of not understanding Conclusion • Use a mixture of methods, especially for complex concepts.

8. Developing a New “Mixed Methods” Tool Based on HAVEG findings, 4 key complex concepts were identified through consultations: • False sense of security • False positive HIV test result • Need for contraceptive • Vaccine cannot cause infection but may enhance susceptibility A new “mixed method” tool was developed. Tested in two studies: • AOU Pilot Study (simulated trial in South Africa and Zambia) • Mixed method vs. T/F questionnaire • Comparative Study (Phase I trial in South Africa and Uganda) • Mixed method vs. Narrative vs. T/F questionnaire

9. Study findings • Reinforced the HAVEG study findings • T/F consistently scored higher than scenarios • Relatively short time to administer • Highlighted issues around counselor capability to score objectively • Remove counseling “hat” • Familiarity with concepts and tool • Real time scoring

10. Refining the mixed method tool True/False statements covering basic informed consent concepts 4 Scenarios covering complex concepts • Each scenario ‘concept’ had 2-3 measures Example: Concept: False sense of security Sub-concepts: • Some volunteers may not be getting study vaccine; they may be getting placebo • The study vaccine may not protect volunteers against HIV infection • Volunteers must not increase their risk behavior Scoring sheet was developed to make the process more objective.

11. Implementation of the mixed method tool • Implemented in Phase I HIV Vaccine trial in Kenya, Uganda and Rwanda • Counselors/doctors trained to administer and score new tool • Scores tallied in real time by administrator • First 6-10 volunteers at each center had an independent scorer at administration for quality control purposes • Based on results volunteer passes or is further educated and re-tested.

12. Staff thoughts prior to training • Initially, staff did not see the benefit of the new tool • Concerns raised about: • Cost of implementing the new tool • Skill level required to implement • Time required to administer • Screen outs • Researchers were also concerned about staff ability to be objective

13. Feedback from Research Center Staff after trial implementation Following training and implementation, feedback suggests that the tool was well liked by both staff and volunteers • Staff felt that the tool allows volunteers to better demonstrate their understanding “Volunteers enjoy the scenarios because they are more interactive and they feel more at ease …an opportunity to exchange ideas” (Doctor, Rwanda) “…the scenarios also helped us realize there is a chance that we had been making assumptions about volunteer understanding for example, a person may know what a placebo is but not necessarily why it is given” (Nurse Counselor, Kenya)

14. Feedback from Research Center Staff on use of tool in a real trial • Staff said that volunteers appeared to be less nervous as compared to when closed-ended tools were used “Scenarios are much better because when you read them the volunteer feels like they are in a conversation while in the T/F they feel like they are doing an exam and get nervous” (Nurse counselor, Rwanda). • The tool allows staff to assess their own understanding of trial concepts and therefore make necessary improvements to informed consent process. “The new tool helped the counselor realize their own understanding of concepts and their ability to explain them well” (Nurse Counselor, Kenya)

15. Feedback Staff on use of new tool in a real trial • The staff felt the design of the tool allowed for clear and consistent scoring “The use of a marking scheme was very helpful to ensuring all information was captured and scoring was correct” (Nurse counselor, Uganda) • Staff also provided suggestions for improving the tool for future use “The volunteers were very nervous during the T/F but relaxed during the scenarios. It may therefore be beneficial to do the scenarios before the T/F” (Doctor, Uganda) “Enhanced susceptibility was…particularly difficult. Consider rewording and ensuring the informed consent session clearly explain the concept” (nurse counselor, Kenya)

16. Summary • Results from studies suggest that open ended questions assess understanding of complex questions better • A mixed method tool is a viable option to use within the context of a clinical trial • In order for the tool to be successfully implemented, researcher commitment to ensure staff are well trained is essential • The process helped counselors realize the gaps in the informed consent process, and new ways of addressing these gaps.

17. Acknowledgment • Clinical trial participants • Clinical research center staff • Bonnie Bender, Frances Priddy, Graham Lindegger, Jennifer Lehrman, Pat Fast, Prince Bahati, Sabrina Welsh, Sagri Singh, Tsedal Mebrahtu