21-Tech Formative Evaluation Phase III Results Presentation

1. 21-Tech Formative Evaluation Phase III Results Presentation CECILIA GARIBAY, PRINCIPAL JANE SCHAEFER, PROJECT DIRECTOR

2. Overview 21-Tech is an IMLS-funded project focused on developing and broadly sharing methods for increasing the capacity of exhibit facilitators to use personal mobile technologies (PMTs) to engage museum visitors in learning. This presentation document provides key results from Phase III of formative evaluation, which focused on testing facilitator training. Formative Evaluation Formative evaluation for 21-Tech involved three phases. Phase I involved an exploratory phase where project partners tested various strategies for PMT facilitation at exhibits. Phase II focused on understanding facilitator-visitor interactions and visitor perspectives of the experience. Phase III formative testing—the focus of this document—assessed the extent to which training successfully prepared facilitators to facilitate PMT-based experiences with visitors. Results were used to improve the training. Phase III Methods Data were collected from June to September 2012. Methods included facilitator surveys, peer observations, and virtual focus groups. Observation data were collected at each partner organization to determine how facilitators were applying concepts covered in the training. Observations consisted of peer-to-peer sessions in which facilitators paired up to observe and provide each other feedback. Twenty-five observations were completed. (See Appendix A breakout of observations by site.) Facilitators completed an on-line survey after having been trained in PMT-based facilitation and had engaged with visitors on the museum floor. The survey was conducted to learn facilitators’ perceptions of and experiences with the training program. Twenty staff members completed surveys at partner sites. (See Appendix A for details by site.) • Phase III Evaluation Questions • To what extent does training increase participating staff’s knowledge of successfully facilitating PMT-based visitor experiences? • To what extent does the training influence participants’ confidence in and motivation toward facilitating PMT-based visitor experiences? • What components of the training seem to be most successful? What aspects are not as successful and how could they be improved? Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

3. Overview, cont’d. • Garibay Group staff also conducted four focus groups (one with facilitators at each partner museum) via video conferencing to collect more in-depth data, probe more deeply into survey findings, and gain increased insight into facilitators’ experiences. • Garibay Group staff developed evaluation protocols and instruments, worked with partners throughout data collection, and analyzed the data. • Facilitation staff at the four partner institutions conducted peer observations, recorded their observations using a form consisting of close-ended and open-ended questions, and submitted completed data to the evaluators. • Data Analysis • Garibay Group, as the external evaluator, conducted all data analysis. Survey data were analyzed through descriptive statistics. Summary statistics were calculated and summarized in histograms and cross-tabulations. Descriptive observation data and open-ended survey questions were qualitatively analyzed and coded using an emergent coding strategy (Altheide, Coyle, DeVriese, and Schneider, 2008). In emergent coding, categories are established after an examination of the data. Limitations One major limitation of this study involved observation data. While data were intended to provide insight on the ways and extent to which facilitators applied skills and ideas learned during training, there was not always enough description of facilitator-visitor interaction to draw clear conclusions. Thus, observation data were primarily used to triangulate findings garnered from the survey data. Garibay Group originally gave this presentation to the 21-Tech partners. This overview section was added to provide context for the reader. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

4. Participating Facilitator Background Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

5. Facilitator Background Most facilitators participating in this study were part-time paid staff or volunteers. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

6. Facilitator Background, cont’d. • Only a quarter of facilitators personally owned a tablet. Thus, most facilitators’ experiences with a tablet came primarily through use on the job. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

7. Facilitator Background, cont’d. • Most facilitators had little work experience in science centers or museums and, in general, little facilitation experience. As a result, less experienced respondents could not deeply reflect on their facilitation practice; most were unable to compare facilitation with PMTs to facilitation without them. Less experienced facilitators focused on technical and practical aspects—for example, readily accessing apps, carrying the tablet, or mastering content. They were not as focused on how PMTs could inform their approach to facilitation and their interactions with visitors. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

8. Facilitator Background, cont’d. • A slightly higher number of facilitators rated themselves as competent or expert in facilitating visitor experiences than in using PMTs as tools for facilitation. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

9. Facilitator Feedback Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

10. Expectations • The majority (85%) of facilitators reported that the training met or exceeded their expectations. Fewer than a third, however, reported that their expectations were exceeded. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

11. Satisfaction with Training • Facilitators were generally satisfied with most aspects of the training; in most areas, 80% or more indicated being “satisfied” or “very satisfied”. Facilitators, however, provided lower satisfaction ratings of the training module explaining how apps related to exhibit content. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

12. Usefulness of Training • At least eighty-five percent of facilitators rated the four training activities as “moderately” or “very” useful. Based on interviews and survey comments, the role-playing activities may have been less useful because not enough time was devoted to role-playing during training. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

13. Learning • All facilitators responding to the survey reported increased knowledge in all aspects covered by the training. Facilitators especially noted learning to approach and engage visitors with the tablet and to successfully facilitate interactions at exhibits using the PMT. Facilitators reported lower ratings in the training’s impact on their knowledge of the tablet and apps. It is not clear, from facilitator comments, why this category received the lowest ratings; facilitators may have already known enough about these areas that the training had minor impact. We will investigate this in the next project phase. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

14. Most Useful Skills or Ideas Learned When asked “What three skills or ideas that you learned from the training were the most useful in facilitating visitor experiences?”, facilitators most mentioned these topics: • Integrating the iPad into an explanation or activity with an visitor"Using the applications made explaining certain things much easier since some ideas are easily understandable with visual aids provided by 21-Tech. I was also more likely to engage visitors of more diverse age groups with different levels of understanding since the applications worked with the majority of visitors.” • Approaching visitors with an iPad“[I learned to] Be as specific as possible when approaching visitors. Be short and sweet. Have your approach be tailored to the age of the visitor.” • Making connections between an exhibit and an app “I learned how to make connections between the apps and the [tablet]. For example, with the Sciencenter colored shadows exhibit and the tablet we were trained well and this helped me facilitate with success. The more specific training on content, the more confident I felt approaching visitors.” • Learning which apps fit a specific exhibit or situation“[I learned] All the different apps that we can use for specific exhibits.” Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

15. Confidence • After the training, 95% of facilitators reported increased confidence in facilitating visitor experiences using PMTs at exhibits. Only one facilitator rated his or her confidence a “3” both before and after. My confidence in successfully facilitating visitor experiences using the tablet at exhibits: Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

16. Equipped to Facilitate with PMTs • Three-quarters of facilitators reported feeling “adequately” or “very” equipped after training to successfully facilitate visitor experiences at exhibits using the tablet. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

17. Facilitator Understanding of PMT Role • All facilitators agreed that PMTs could enhance visitors’ learning, with 80% providing “strongly agree” ratings. It is intriguing to find such high agreement by facilitators on this issue. In the next project phase, we will further explore this area to learn how facilitators view PMTs as enhancing visitor learning. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

18. Motivation to use PMTs • Training appeared to successfully motivate facilitators to use PMTs with visitors. Nearly all reported being motivated “a lot” (45% of respondents) or a “great deal” (50%) to use PMTs in facilitating visitor experiences. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

19. Value Professionally • All facilitators agreed that PMT-facilitation training was professionally valuable. Responses were almost evenly split, with 45% reporting the training to be “moderately valuable” while 50% called it “very valuable.” This split suggests that the training could be improved. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

20. Improvements Facilitators suggestions to improve the training included providing more examples, more time to role-play, and more time on the floor to practice using the tablets. More opportunities for role-playing Levels of training based upon facilitators’ previous experience with tablets More examples of successful interactions (e.g., videotaped interactions) More examples of ways to link exhibits to apps More time to practice Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

21. Challenges Identified Integrating certain exhibits with apps on the tablet was difficult, but the applications themselves demonstrate abstract ideas that may be challenging for a visitor to visualize. Some apps only had slight relevance to the exhibit, and it would require taking explanations in a specific directions to have the app be helpful. There really needs to be more time exploring and getting comfortable with ALL of the apps….Also, some exhibits did not even have apps, or the quality of apps was so low that they would never—and will never—be used. Feeling confident with the content [is important]! It would be useful to have more background information on the content. The more completely training can simulate actual use of the tablet with visitors, the better it will be! [I wish] we had more time to experiment with exhibits, and to train on specific apps, or even if we had more apps. Fit between app and exhibit Facilitators reported challenges in making apps fit the exhibits. For example, they may have been given an app they thought to be only tangentially related to an exhibit. Children competing for table Visitors, especially children, sometimes struggled to share tablets. Facilitators sometimes found themselves having to mediate between siblings all vying for time on the tablet. Crowded or loud exhibits Not surprisingly, facilitation in crowded or loud exhibits was more difficult than in less crowded or quieter exhibits. In one overly crowded exhibit, for example, a facilitator said, children competed to grab the tablet. Feeling confident with exhibit content or app It takes time for facilitators to feel confident with exhibit content or an app. While the trainings were comprehensive, the main issue was often ensuring that each module was long enough, particularly role-playing practice time and time on the floor with visitors. Beginning and ending interactions Some facilitators found the most challenging aspect of facilitation to be either approaching visitors to begin an interaction or ending an interaction comfortably. Sometimes visitors liked the tablet so much, they wanted to stay and play on it. Other times, a child might have felt she was “done” with the interaction before the facilitator was able to end it effectively. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

22. Recommendations Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

23. Recommendations • Main Training Topics • One key finding that emerged is that training must address five main topics: • General facilitation skills: for instance, how to talk to visitors and how to engage with questions • Exhibit content including the science underlying the exhibit • Tablet hardware • Specific apps, including how the apps are organized on the tablet and—most importantly—how the apps relate to the exhibit • Specific facilitation skills for tablet interactions; for example, how to approach visitors with the tablet, how to help visitors share a tablet, and how to end an interaction. This training needs to include role-playing with other facilitators or staff before trying the techniques with visitors. • Training Paths • One consideration in training is to understand the relative experience of the facilitator and provide a training track appropriate to the individual’s experience. • Based on the data, we identified three different training paths based on the three types of facilitators: • Novice Facilitator • Tech-Savvy Facilitator • Experienced Facilitator • The graphics on the following pages illustrate each of the three paths. Structure and Time of Training The 21-Tech training can be divided into three types: 1) classroom time, i.e., traditional training in which the content is presented to facilitators; 2) time on the floor to practice what was learned; and 3) reflection time for facilitators to look back and critically consider their interactions with visitors. Based on the data, facilitators felt they needed more time to practice on-the-floor facilitation with tablets with other facilitators as well as visitors. It also appeared that, in most cases, adequate reflection time was not built into the training structure. Because of these two factors, facilitator training should not be structured as a one-time training, but as a process with several cycles. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

24. Facilitator Training: 3 Tracks Novice Facilitator General facilitation training is essential. Tech-savvy Facilitator May not need general training on the iPad hardware. Experienced facilitator Despite experience, needs time to role-play and practice facilitation with the iPad. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

25. Reflective Practice Phase III findings also have implications for facilitators’ ongoing reflective practice. Ideally, following training, facilitators would use with visitors what they learned on the floor then discuss their experience with other facilitators; collaboratively, they would refine their approaches, choose new apps for exhibitions, and/or choose new exhibitions for which to add facilitation. They would then test new facilitations through role-playing before using them with visitors, re-starting the cycle. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

26. References and Appendices Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

27. References Altheide, D., Coyle, M., DeVriese, K., & Schneider, C. (2008).  Emergent Qualitative Document Analysis.  In Handbook of Emergent Methods. Hesse-Beiber, S.N. and Levy, P., eds., NY, NY: Guilford Press. Miles, M. B. & Huberman, A. M. (1994). Qualitative Data Analysis (2nd ed.). London:  Sage Publications. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

28. Appendix A: Data Sources by Site The table below details data sources by institution. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

29. Appendix B: Training Plans • CMH • Introductory Session (2 hours minimum) • Discussion of what 21-Tech is, who our partners are, goals of the program, and the vision for CMH • Introduction to the iPad: describing features, tips, tricks, how to use basics • Explanation and examples of the different categories of apps (content construction, skill development, and experimentation/exploration) and the app-exhibit connections for each. • Individualized exploration of apps, getting used to iPads, and Q/A regarding iPads and apps • Discuss how to use apps to lengthen, extend and deepen visitor experiences. Also discuss how NOT to approach visitors with an iPad. • Individual time to explore apps and discover the app-exhibit connections. • Visit Kidropolis and demonstrate (or role-play) how to make the app-exhibit connections when interacting with a visitor. Visit the Vet Clinic, Diner, and/or Bank. • App choices: iWashDogs, iPizza, Big Fork Little Fork, BillsCoinsDrills, Jungle Math • Feedback from trainer, discussion and reflection • Content Connections and Skill Development Training (1.5 hours minimum) • Discuss new apps facilitators have discovered/used since the last training. • Allow facilitators time to explore the Invention Convention apps: Paper Plane Project, World Record Airplanes, and Wind Tunnel. Discuss the approach for using these apps. Allow facilitators to suggest different approaches/uses. • Visit Invention convention and practice making paper airplanes. Also, practice using the Wind Tunnel app in the gallery. • Feedback from trainer, discussion and reflection • Experimentation/Exploration Facilitation Training (1.5 hours minimum) • Review the apps which act as tools to help kids explore and experiment in various exhibits. • Allow facilitators time to explore the iMotion HD app, and how stop motion works. Allow facilitators time to experiment with the app. When done, discuss where/how the app may be used. Discuss other apps which work as tools, such as the Giant Timer and Camera. • Practicum time facilitating visitor experiences, using iMotion HD, Giant Timer and/or Camera. • Feedback from trainer, discussion and reflection • **If needed, schedule follow-up training (1.5hrs) to provide facilitators more time to practice interacting with visitors using the iPad. • Final Practicum (1.5 hours minimum) • “Solo” experience. Assess ability to utilize the facets of the iPad to enhance visitor experience. Determination if training is completed.. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

30. Appendix B: Training Plans, cont’d. • NYSCI • Day 1 • Goals • Introduce Explainers to the 21-Tech Project • Expose Explainers to the various digital tools used in 21-Tech • Objectives • Explainers will be able to navigate and operate through the iPod, IPad2, and the Motorola Xoom • Intro to 21-Tech Project • How to use the Xoom • How to use the iPad/iPod (iOS) • Scavenger Hunt (Two groups. Each group takes a device to complete) • Day 2 • Scavenger Hunt 2 (Same groups as previous day. Each group use a different device than previous day) • Peer Training • Take turns explaining the exhibit to each other to gain a deeper understanding. • App review • Spend time going through each app • Day 3 • Peer Training (Explainers explain exhibits they were assigned, incorporating PMTs into their facilitation). • Reflect on the day/training Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

31. Appendix B: Training Plans, cont’d. • OMSI • This training doesn’t give them everything they need to use iPads on the floor, but it does get them familiar with the iPad, and how we expect it to be used. Next steps after training include checking in with educators in their work area, and becoming familiar with area specific apps. • DEMO Objective/ Bower Birds 20 minutes • The training opens up with the instructors facilitating two iPad demos as examples. • INTRODUCTION Welcome 10 minutes • Individual introductions. Name, work area, familiarity with iPads. • Introduce the 21-Tech grant; Why we have the iPads, how we intend to use them. • The goals and limitations of this training.   • INSTRUCTOR LEAD ACTIVITYMeet the iPad45 minutes • Introduction to the iPad where volunteers will be introduced to the basic functions and features • INSTRUCTOR LEAD ACTIVITYSwitch iPads20 minutes • Volunteers switch iPads and repeat all of the functions so they become familiar with how to use them, even if iPads are organized differently. • INDIVIDUAL ACTIVITY Break/Free explore 20 minutes • INSTRUCTOR LEAD ACTIVITY App Races10 minutes • This is time to apply what they’ve learned. Give the group a challenge and see how long it takes for all of them to meet that challenge. • DEMOProscope5 minutes • GROUP DISCUSSION Rules of the Road and Informal vs. Formal Education15 minutes • How is what we do here at OMSI (informal education) different than formal education? • How should we hold/ present the iPad? • How do you decide if it’s an appropriate time to use the iPad? • When should the iPad be used on the floor (with visitors)? • For what should the iPad be used? • INSTRUCTOR LEAD ACTIVITY Next Steps5-10 minutes • Discuss successes and learning’s • INDIVIDUAL ACTIVITY Demo Design Plan20 minutes • Discuss different ways the iPad may be incorporated into work areas or explore apps. Participants should leave with an idea of a specific exhibit or demo they want to work with. Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013

32. Appendix B: Training Plans, cont’d. • Sciencenter • Intro to 21 Tech (30 min) • Review 21-Tech project • 21-Tech Factsheet • Review Sciencenter’s tips for interacting with visitors and facilitation strategies for 21-Tech exhibits: • Training Tips • Five-Step Process • Maddie’s YouTube video • Learn to use each of the 21 Tech tools: iPad and iTouch • Free exploration with technology and tools • Exhibit Training (30 min) • Introduce and review background materials for exhibit components and discuss science content(ex. Colored Shadows): • Colored Shadows Exhibit Guide • Colored Shadows Prompts • iPad diagrams, images, and photos • Introduce and practice using the apps: Bobo and Color Uncovered • Practice facilitation with another volunteer or staff person • Reflect on the Experience • Discuss what works and what doesn’t • How do you initiate an interaction (what is the hook)? • How do you incorporate PMTs into the interaction? • How do you transition from exhibit to tool and back to the exhibit? • How do you connect content to visitors’ lives? • How do you use with younger audiences or older audiences? • What are the differences in working with families and school groups? • Recommendations and suggestions Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013