21-Tech Summative Evaluation Report

21-Tech Summative Evaluation Report Cecilia Garibay & Jane Schaefer February 2014

Table of Contents Project Overview3 Evaluation Design and Methods 5 Results Levels 1 and 2: Reaction and Learning10 Level 3: Transfer 15 Level 4: Impact 26 Impact on Facilitators 27 Impact on Visitors 31 Impact on Partner Organizations 35 Dissemination 44 Conclusions and Lessons Learned53 Conclusions 54 Lessons Learned 56 References57 Appendices59 CreditsChildren’s Museum of Houston: all videos + cover, 10, 30, 37, 38, 44, 53, 57, 59 Garibay Group: 45 New York Hall of Science: 5, 40 Oregon Museum of Science and Industry: 26, 39 Sciencenter: 15, 33, 41 Garibay Group | 21-Tech | Summative Evaluation | February 2014

Project Overview 21st Century Exhibit Facilitators and Personal Mobile Technologies (21-Tech) was a project of the Children’s Museum of Houston (CMH), New York Hall of Science (NYSI), Oregon Museum of Science and Industry (OMSI), and Sciencenter (SCI), funded by the Institute of Museum and Library Services(IMLS). The goal of 21-Tech was to develop and broadly share methods for increasing the capacity of exhibit facilitators to use personal mobile technologies (PMTs) to engage museum visitors in learning at the museums. Garibay Group was contracted to conduct the project evaluation. This report focuses on summative evaluation results. This initiative explored the use of PMTs as part of facilitated visitor experiences intended to enhance and advance hands-on learning. Rather than developing mobile applications, 21-Tech gathers learning and experience scaffolds through open-source apps and other resources that allow facilitators to customize the exhibit experience for visitors. (For the sake of brevity, we use the term “apps” to refer to both applications and resources such as photos and video throughout this report.) Significant shifts in technology adoption have taken place since the start of 21-Tech in 2011. The accompanying figure shows the rapid growth of tablets and smartphones since 2011, a rise projected to continue through 2017. By 2012, it became clear to the 21-Tech team that tablets would be the best devices for facilitators to use with visitors due to the wider adoption of tablets, the availability of appropriate apps, and facilitators’ positive experiences with the devices on the floor in terms of the screen size, usability, and portability (Garibay and Schaefer, 2011). Garibay Group | 21-Tech | Summative Evaluation | February 2014

Project Overview, cont’d. 21-Tech involved partners from diverse types of organizations in terms of size, staffing, and location. Table 1. Partner sites with numbers of facilitators involved in 21-Tech and training duration. Source: Partner statistics, Fall 2103. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Evaluation Design and Methods Garibay Group | 21-Tech | Summative Evaluation | February 2014

Evaluation Design and Methods Formative Evaluation Background Formative evaluation with iterative prototype testing guided the project’s development. Three phases of testing—each building on each other—were included. Phase I focused on investigating a broad range of questions about how facilitators might use PMTs to positively engage visitors in various exhibits. For example, the team explored such questions as: “What kinds of exhibit interactions work the best, for instance, observing, hands-on, etc.?”; “How can we use the PMT in ways that are different from other extension activities?”; “What about the PMT makes for a special or unique kind of interaction?” Phase II focused on testing the same apps across partner sites at exhibits commonly found in many science centers and museums (e.g., exhibits about colored shadows, structures, microscopy, pitch). This allowed us to test whether earlier findings would be replicated at other sites and also ensured that the apps worked at different types and sizes of museums. In this phase, we also focused on obtaining data on the nature of facilitator/visitor interactions (Garibay and Schaefer, 2012). Phase III centered on assessing the training curriculum developed by partners and the extent to which it successfully prepared staff to facilitate PMT-based experiences with visitors(Garibay and Schaefer, 2013). • Summative Evaluation • Summative evaluation (the subject of this report) focused on assessing the project’s outcomes. As a 21stCentury Museum Professionals-funded project, the project primarily focused on professional development—that is, increasing museum staff capacity in using PMTs in their facilitation initiatives. • Project Outcomes • The 21-Tech project had the following three outcomes: • Project team leaders increase the capacity/infrastructure of their museums to advance exhibit facilitator use of PMT in exhibit facilitation. • Exhibit facilitators at partner museums continue to increase their ability to use PMT to engage visitors in learning. • Other museum professionals increase their understanding/use of PMT in exhibit facilitation through the project findings. Definitions of Key TermsPersonal Mobile Technologies (PMT): A handheld or personal device or related software that can link to the Internet. These technologies include smartphones, PDAs, handhelds, media players, tablets, and netbooks. Tablets: The term tablet in this report encompasses both iOS and Android devices. 21-Tech primarily used iPads and iPadminis, though a few 7-inch Android tablets were also included. Apps: For the sake of brevity, we use the term “apps” to refer to both applications and resources such as photos and video. Exhibit facilitator: A paid staff member or volunteer whose primary role is to engage visitors in learning through exhibits. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Evaluation Design and Methods, cont’d. • Kirkpatrick’s (1998) widely used model for evaluating professional development and training has been adapted to various fields. The model outlines four levels of evaluation, with each successive level building on information from the previous ones. Evaluation always begins with level one and moves sequentially through the other three levels. Each successive level represents a more precise measure of professional development effectiveness but requires more rigorous and time-consuming processes. The accompanying table summarizes the focus and key question of each level. Guskey (2000), who adapted the Kirkpatrick Model to evaluate educator professional development initiatives, argued for adding another stage between levels 2 and 3 to measure organizational support, because it is essential that organizational structures support one’s professional development and practice. Yet in Garibay Group’s experience evaluating professional development initiatives, organizational aspectscannot be confined to a discrete stage. Organizational factors are, rather, critical contextual factors that can ultimately influence impacts. • This report is organized according to Kirkpatrick’s four levels. We first discuss reactions to training and involvement in 21-Techand learning, then transfer, and, finally, evidence of impact. • Table 2. Kirkpatrick’s model for evaluating professional development (1998) Garibay Group | 21-Tech | Summative Evaluation | February 2014

Evaluation Design and Methods, cont’d. Methods This study used a mixed methods approach (Green and Caracelli, 2003), collecting quantitative and qualitative data. Data were collected at all four sites: CMH, OMSI, NYSCI, and SCI. Collecting key information about the same constructs in different ways allowed us to seek convergence or corroboration of the information to confirm and/or further explain the outcomes (Greene and Graham, 1989). Data were collected from November 2012 to November 2013. Methods included project leader interviews, facilitator focus groups, facilitator surveys, visitor surveys, observations with follow-up interviews, video observations, webinar and workshop surveys, and website analytics. Surveys Facilitators completed an on-line survey after having been trained intablet-based facilitation and having engaged with visitors on the museum floor. The survey was conducted to learn facilitators’ perceptions of and experiences with 21-Tech, particularly how it impacted their professional development. A total of 31 facilitator surveys were collected, and the response rate was 94%. (See Appendix A for details.) After interacting with a facilitator who had a tablet, visitors were randomly surveyed about their experience. (Surveys were administered by museum staff trained by Garibay Group.) These surveys were intended to gain understanding of visitor perceptions of their interactions with facilitators using tablets and of what visitors were taking from the experience. A total of 152 facilitator surveys were collected.The response rate was 92%. (See Appendix B for more details.) Observations & Intercept Interviews Observation data were collected at each partner organization to understand the nature of the visitor-facilitator interaction. After the observations, data collectors conducted short intercept interviews with the visitors to better understand how they perceived the interaction and what they took from it. In total, 39 observations and interviews were completed. (See Appendix C for more details.) Video observations were also conducted at CMH to triangulate observation data and provide a visual record of interactions. A total of 6 observations were videotaped at CMH. Facilitator Focus Groups Garibay Group staff conducted four focus groups (one with facilitators at each partner museum) via phone or videoconferencing to collect more in-depth data, probe more deeply into survey findings, and gain increased insight into facilitators’ experiences. A total of 15 facilitators participated. Project Leader Interviews Project leader interviews were conducted at each project site in order to understand 1) the project design and implementation and 2) 21-Tech’s impact on both leaders’ and facilitators’ professional development and on the organization as a whole. Nine leaders in all were interviewed. Webinar & Workshop Questionnaires In order to disseminate findings to the larger museum community, the 21-Tech team held a number of workshops, webinars and forums. Two workshops—one held prior to the Association of Children’s Museum (ACM) Annual Conference in May 2013 and another before the Association of Science and Technology Center’s Annual Conference (ASTC) in October 2013—and one ASTC-sponsored webinar held in September 2013 were evaluated. After each of these sessions, participants were asked to take an on-line survey about their experience. The purpose was to assess the dissemination of the 21-Tech findings tomuseum professionals outside the four partner sites. A total of 40 workshop surveys (response rate of 71%) and 35 webinar surveys were collected (response rate of 40%). (See Appendices D and E for details.) Website Analytics Google Analytics was used to measure the extent to which the 21-Tech.org was visited as well as the number of unique visitors, the average number of pages visited, and the average length of a visit. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Evaluation Design and Methods, cont’d. • Data Analysis • Garibay Group, as the external evaluator, conducted all data analysis. Survey data were analyzed through descriptive statistics. Summary statistics were calculated and summarized in histograms and cross-tabulations. Though facilitator and visitor survey data were disaggregated by site, we found no significant differences. Therefore, this survey data are typically presented in aggregate. • In most cases, we present survey responses in percentages. (Some percentages do not add up to 100% due to rounding.) Where more appropriate, the actual number of responses (n) is provided. • Descriptive data, including interviews, observations, focus groups, and open-ended survey questions, were qualitatively analyzed and coded using an emergent coding strategy (Altheide, Coyle, DeVriese, and Schneider, 2008). In emergent coding, categories are established after an examination of the data. In some cases, open-ended responses mentioned more than one topic. As a result, the number of responses may add up to more than the number of respondents. Limitations While it would have been desirable to have a comparative design of exhibit experiences with and without tablets, budget constraints and the timing of the summative evaluation did not allow for this type of study design. Evaluation used a mixed-methods approach. Collecting key information about the same constructs in different ways allowed us to seek convergence or corroboration of the information to confirm and/or further explain the outcomes (Greene, Caracelli, and Graham, 1989). . • Figure 3. Visitor-Facilitator Interaction Model Early in the testing and development of the 21-Tech project, Garibay Group developed a model to represent the different factors that affect a visitor-facilitator interaction and impact the visitor and the facilitator. These factors include the facilitation skills of a facilitator, the PMT, which is made up of the device itself (e.g., the iPad, Nexus 7, iPhone, etc.) and the apps used on the device, and the exhibit at which the facilitation is taking place. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Results Levels 1 and 2: Reaction and Learning Garibay Group | 21-Tech | Summative Evaluation | February 2014 10

Perceptions of Training • Facilitators surveyed reported robust increases in their learning following the training. These learning increases included increased knowledge of tablets and apps, how to better approach and engage visitors, and ways to facilitate with exhibits using a tablet. Respondents did indicate, however, that while training increased their understanding, they needed additional time putting these ideas into practice. • Table 3. Facilitator ratings of increases in learning after training • I find that practice on the floor with the iPads is the most beneficial part of training. Theory is a great part of the training, but it cannot replace the value of actual trial and error. Each person has different techniques that can only be honed/discovered as facilitation is practiced. Also, team collaboration and reflection with other facilitators help to prompt new ideas and [are] a great way to learn how iPads can be used in a variety of ways. • The most useful part was the sharing and feedback of other 21-Tech tablet facilitators. Getting feedback from others and sharing my own personal experiences with others has helped the most. Also the approach for using the tablets with visitors was very useful. There are more ways to introduce the iPad instead of forcing it into the visitor’s face. • I feel that the most useful part of training was being able to talk to the lead 21-Tech managers and being able to give them our input. The fact that they listened and helped adjust our issues, concerns, and ideas was amazing. n = 21 Garibay Group | 21-Tech | Summative Evaluation | February 2014

Increase in Confidence When asked to reflect on their entire involvement in the 21-Tech project, facilitatorsindicated significantly higher levels of confidence and feeling equipped in facilitating visitor experiences using a tablet. For the statement “I feel confidence in successfully participating visitor experiences,” the “mostly true” and “very true” agreement ratings increased from 26% at the start of the project to 97% at the end of the project. On feeling equipped, the “mostly true” and “very true” agreement levels increased from 36% to 94%. n = 31 n = 31 Garibay Group | 21-Tech | Summative Evaluation | February 2014

Facilitation Skills and Ideas Learned Facilitators were asked to list three important skills or ideas they learned about by participating in 21-Tech that they had applied in their work. While facilitators listed a range of skills and ideas, three were most often mentioned: how to use apps with an educational intent (including learning the limitations of using tablets); learning general facilitation skills related to interacting better with visitors; and learning how to customize the experience based on the age or interest of a visitor group. • One important skill I obtained from participating in the 21-Tech project was learning how to use the different apps in different galleries with different age groups of children. • The first [skill] would be using the resources available to you to aid visitors' experience. The second would be taking the concepts of the galleries into greater consideration to properly attach a theme to the use of the tablets. The third would be on-the-spot thinking on how to further engage a visitor in a particular exhibit or activity. * “Other” included: visitors enjoyed the iPad, children know how to use tablets, and collaborating with other staff Garibay Group | 21-Tech | Summative Evaluation | February 2014

Motivation to Use Tablets in Facilitation • Training and experience on the floor appeared to successfully motivate facilitators to use tablets with visitors. Participation in the project also appeared to significantly increase motivations to use tablets in facilitation. After participating in 21-Tech, 55% more respondents indicated that the statement “I am motivated to use a tablet in facilitative visitor experiences” was “very true” than had done at the beginning of their involvement. I use the technology more now than I did at the beginning….it was a desire for me to want to continue to add more ways in which visitors could interact with the material in my lab. And the iPad was there and it was a quick and easy way for me to be able to do that….It was the demo stuff first, and then expanding into classes, and now I also use it as a way to facilitate the 3-D printer. n = 31 Garibay Group | 21-Tech | Summative Evaluation | February 2014

Results Level 3: Transfer Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer The 21-Tech project’s visitor-facilitator interaction was not a simple method to be taught once and immediately executed successfully on the floor with visitors.Therefore it was the process of trying to apply what they had learned from training that helped facilitators become more comfortable with and hone their skills using PMTs.The more that facilitators tried apps at exhibits, shared with other facilitators, and reflected on the process, the more they deepened their skills and practice. Facilitators shared various aspects of PMT-based facilitation they had to master: Learning the Technology Facilitators found that an immediate need in the process was becoming comfortable using the tablet and apps. While a small number had experience with tablets before their involvement with 21-Tech, most had to learn the basics of how to navigate, open, and interact with apps on a tablet. App organization on the tablet proved to be a challenge. Initially, most sites did not have the apps arranged in the best way for facilitators. For instance, apps may have been organized into folders by concept. After struggling with trying to find the appropriate app on the floor, facilitators learned that organizing apps by galleries was usually the most efficient way. Understanding the Apps Facilitators were initially challenged in fully understanding the apps themselves and then how to use them within the context of a specific exhibit (or in the case of OMSI, a lab, class, or hands-on demonstration). Although some aspects could be covered in training, most facilitators found they needed more independent time exploring and learning the app themselves. Role-playing with other facilitators helped them determine and refine how to use the app within a specific exhibit or demonstration. Integrating the Tablet into the Facilitation Facilitators also had to determine appropriate ways to integrate the app into their interactions with visitors. Initial concerns were that this would be awkward or that the tablet would take the visitors’ attention from the exhibit itself. While training and modeling by project leaders helped facilitators understand effective ways to integrate apps in the visitor experience, each facilitator ultimately had to determine the best ways for him or herself. With practice, a facilitator refined his or her ability to introduce the tablet, smoothly transition between the tablet and the exhibit, respond to visitor questions, and end the interaction. Reflecting on Practice Facilitators repeatedly emphasized the importance of sharing experiences with other facilitators as part of their learning process. Communication and sharing gave facilitators opportunities to hear about other interactions, facilitation scenarios, and solutions to challenges that they themselves might not have encountered. It also promoted group problem-solving that made the 21-Tech project both more efficient and more valuable to facilitators. Based on interviews with team leaders, in some organizations (particularly CMH), this group problem-solving model has the potential to be applied to future projects and initiatives well beyond 21-Tech. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. In time, facilitators became more experienced using tablets with visitors at exhibits. As they became more proficient with the apps and tablets, facilitators were better able to customize interactions with visitors based on a group’s needs. For example, the figure on the right illustrates the ways in which facilitators developed skills in using apps to customize the experience for visitors.Early in their involvement with the 21-Tech project, facilitators might only have been comfortable using one designated app with an exhibit. As their experience and comfort with the tablet-based facilitation increased, facilitators could better select the best app for an interaction from a group or “suite” of apps that they knew worked well with the exhibit. For instance, a facilitator talking with a young child might choose a different app from the group of apps than if working with an older child. As their experience and comfort increased, facilitators were able to move or “jump” between the apps depending on a question or interest of an individual visitor. A facilitator, for example, might start the interaction with an app that showed the human skeleton. When the visitor asked a question about the organs protected by the ribcage, the facilitator could respond to the question and continue the discussion by “jumping” to an app that showed how the heart works. Figure 8. Model of changes in use of apps by facilitators as proficiency increased Jumps between apps in a “suite of apps” Proficiency with tablet and apps Chooses an app from a “suite of apps” to use with exhibit Uses one app per exhibit Time Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Figure 9. Model of reflective cycle practice In this video, a facilitator responds to a boy’s question by “jumping” to a different app that helps answer his question. To play, click http://youtu.be/ZpO95RinUtQ. This graphic shows the reflective cycle practice that emerged as part of evaluation findings at the formative stage. While training was an important component of the preparing facilitators, successful interactions and developing facilitators’ capacity required an ongoing cycle of reflection. A similar reflective process operated at the team leadership level, where feedback from experimentation and evaluation guided discussions and reflection on subsequent stages of the project’s development. [Facilitators] would come up to me all the time and go up to each other: “Hey, guess what? You know how we used this app? …Well I did that, but I also added these two apps.” So that’s how we discovered the app jumping, where it was…taking someone else’s idea but then adding on your own spin on it. So I think that’s where we discovered real growth….She [A facilitator] said that “as 21-Tech grows, we feel that we grow.” Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Observation data provided additional ways to assess the extent to which facilitators could effectively use tablets in their interactions with visitors. Facilitators effectively engaged groups with the physical exhibit, the tablet, and in direct conversation during their interactions. Note the high levels of all three types of behaviors. Table 4. Interaction Times Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Facilitators also successfully engaged both children and adults in groups. As might be expected, children were nearly always involved actively in the visitor-facilitator interaction. In most cases (75% of interactions), adult caregivers were also observed engaging(either actively or passively) in the interaction. In 21% of interactions, caregivers not only engaged in the interactions, but also guided and encouraged their children. Table 5. Percentage of adults and children engaged in behavior In addition to behaviors that indicated engagement during visitor-facilitator interactions, observers also noted group members who appeared unengaged. Unengaged visitors were more often adults than children. Descriptions of unengaged adults indicated that they often were watching smaller children in the space or stood to the side observing the interaction. Note: percentages are calculated based on the total number of individuals engaging in the behavior. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Field Notes The 9-year-old boy arrived at the [Bounce Back] exhibit first and cycled through in a counterclockwise direction, bouncing the ball on each floor material. His grandma watched from the bench. The facilitator arrived after a minute or two, and started asking questions. The 7-year-old boy arrived. The facilitator engaged with both boys at [the] exhibit and interpretive panel. The mother arrived, watched, and engaged with boys and the facilitator. The facilitator introduced the tablet app 7 minutes into the interaction. The facilitator and the boys tested different surfaces with the app, led by the facilitator. The facilitator stepped back and the boys continued to play with ball at the exhibit (no tablet). The 7-year-old boy was the last to leave. The mother stood back with a female friend. The 8-year-old girl was excited to see the [Colored Shadows] exhibit and then the [Bobo Explore Light] app…A 13-year-old boy was initially part of the interaction, but went off by himself. A young boy entered the interaction and enjoyed all of it with the girl, but he was not connected to the larger family group. Even very young children, like the girl in this video, were able to be actively engaged in the activity with the facilitator using a tablet if the app was age-appropriate. Notice in this video, two visitor groups with a wide range of ages engage in the interaction. To play, click http://youtu.be/uW_pN83kQzE. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Finding the right “fit” between app and exhibit was critical. The ultimate selection of apps was one that allowed facilitators to move visitors smoothly through experiences, between the tablet and exhibit, in a way that felt integrated. The majority of interactions observed indicated a seamless—or at least fairly smooth—switch between app and exhibit. During formative testing we learned that some apps could be tooenticing and drew visitors away from the exhibit. In other cases, the connection between the app and the exhibit was unclear and left visitors confused. Sometimes the technology got in the way if an app was too involved or complex. It was also important to consider group dynamics to ensure that experiences worked for everyone in the visitor group. Apps that worked well during facilitated interactions had the following characteristics: 1) they had a tight relationship to the physical exhibit that was easy for the visitor to understand; 2) they were simple enough for visitors to quickly figure out how to use them; 3) they were not so compelling as standalone apps to take away visitor attention from the exhibit; and 4) they worked for multi-aged social groups. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Field Notes The facilitator used the iSeisometer [app] to show how different lines were generated and then had 8-year-old boy hold the iPadto move it and demonstrate lines on the different axes. Then the facilitator asked what lines would move when exhibit was turned on. The iPad was in the background, but the boy was equally engaged with what would happen with the structures as with what was happening on the iPad. The children took turns using the microscope and watching and listening to their mother use the iPad, telling them all about blood cells. When asked to compare what they saw from both blood samples, it was seamless going from the microscope to iPad. The facilitator strengthened the connection between the app and the exhibit by physically blocking the red light of the [Colored Shadows] exhibit after the group had turned down the red slider on the [Nexus 7] app. This video is an example of the typical way in which facilitators integrated the tablet into the visitor experience. A child starts with the physical exhibit. He then is introduced to a tablet activity by the facilitator who uses it to begin a discussion about healthy foods with the boy. To play, click http://youtu.be/tYxCUPNOSJQ. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. While the 21-Tech evaluation focused on assessing professional development outcomes, visitor feedback helped us understand how successfully facilitators effectively transferred what they learned to their interactions. Overall, visitors indicated that they enjoyed interactions with facilitators using tablets, with the majority of those surveyed (72%) citing a “very enjoyable” level. It was really fun. It helped me learn a lot of things. —Child The kids seemed very enthusiastic and interested. They play with similar applications at home all the time, so it was nice for them to delve deeper into the exhibit since they are so interested in technology. —Adult It made the experience more fun because we got to see her learn and succeed at making a model. —Adult It's great that she could use that [the tablet] by herself without us. It's natural for her and lots of kids to use that stuff. —Adult It connects something visually with the tablet, which they enjoy using. —Adult [It was] Awesome! Extremely helpful! The magnification was very useful. —Adult I thought that it helped to explain the ideas more quickly. It took less time explain things. —Adult Garibay Group | 21-Tech | Summative Evaluation | February 2014

Transfer, cont’d. Feedback also indicated that a large proportion of visitors would seek to repeat a table-based facilitated experience. More than half indicated that after their initial experience with facilitators using tablets, they were “very likely” to look for museum staff with a tablet at an exhibit. [The tablet] Made it more like real science—testing, re-testing, and building new structures. —Adult Their [the kids] minds are more computer-oriented than paper-reading-oriented ….[Seeing] molecules in 3-D and in motion enhanced the experience. —Adult The exhibits won't change, but technology does. And this adds depth and something new to the exhibit experience. —Adult We never spend this much time exploring the colors and shadows. We usually just walk by, but this time, we went back and forth between the wall and the iPad. It was just so interesting. —Adult Excellent, really good learning tool. Staff member showed kids what went wrong and right and how to make it better. —Adult Garibay Group | 21-Tech | Summative Evaluation | February 2014

Results Level 4: Impact Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Facilitators Nearly all facilitators (97%) surveyed indicated that the 21-Tech project was “moderately” or “very” valuable to them professionally. When asked to describe the most valuable thing about being involved in the project, the three most common responses were: 1) being able to enhance visitor experiences or learning; 2) being exposed to new technology; and 3) increasing their skills or confidence in facilitating. Table 6. Facilitator responses to most valuable aspect of 21-Tech Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Facilitators, cont’d. • Facilitators shared in depth the ways in which being involved in 21-Tech impacted them. • Deepened Facilitation Practice and Helped Customize the Visitor Experience • Facilitators reported that learning to use tablets and apps helped them better customize the experience for visitors and engage them regardless of their age or experience. With tablets, facilitators found it easier to follow up on visitor questions or interests. They could also address different levels of understanding. • For example, one facilitator noted that showing a time-lapse video of decomposition to young children was a helpful way of talking about the concept in addition to showing them an actual decomposed item. Another example was an exhibit about what is “alive” that asked if a seed is alive or not. The facilitator found that children had difficulty understanding the concept, but an animated video showing seeds germinating helped him explain it better. • At CMH, iTranslate was sometimes used to communicate more effectively with non-English speakers. • Facilitators sometimes used tablets to position the children in the group as leaders, since children often were comfortable with • the technology and could then teach the adults in their group. Thus, the children obtained a feeling of achievement. • Some facilitators also commented that using the tablets allowed them to improve the exhibits themselves. For instance, one facilitator noted that he found that some apps containing more current science content could be paired with older exhibits that were somewhat outdated (due to new scientific discoveries). In another instance, a facilitator explained how he used apps to integrate and move through exhibits. • Facilitators also named a number of other ways they used tablets, including to: • show actual examples of phenomena instead of relying on verbal descriptions • demonstrate to visitors how to do research by accessing and filtering information • provide options for activities to do outside of the museum or science center that link to the concepts being discussed • model for families how to use apps for education as opposed to using them only for games The iPad is an attention getter and there is more variety with the iPad. With [a physical prop such as] flashcards, you only have one little thing. We can always google it with iPad to answer kids’ questions when they ask a lot, as oppose to [telling them] “I don’t know.” [The most valuable part of 21-Tech was] Learning how to use a tablet to access information that supports information and concepts that I explain to visitors. Aside from the functional value, I think it is valuable for visitors especially kids to see that tablets can be used for educational purposes. Technology draws something out of visitors. The kids love it….technology draws their attention, breaks down barriers of different backgrounds and languages. We have an entire exhibit area dedicated to light….I used that one app on one exhibit where it would show you how you mix different colors. Then, as we moved on to different exhibits in the light area,…I started talking about intensity… [then] we went to another exhibit, explained that, and then it just worked out well that there was also another app that would facilitate that exhibit. [The most valuable part of 21-Tech was] The experience of engaging so many different visitor groups and learning to explain the science behind different exhibits. This was a new kind of teaching for me. I can connect [the visitors] to the tablet first and then the exhibit. It helped me gain more confidence in myself and made the exhibits more enjoyable. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Facilitators, cont’d. Less experienced facilitators, on the other hand, talked about how 21-Tech helped them become more confident about approaching and engaging visitors. One shared, for example, that before 21-Tech, he felt the exhibits did “everything” and that he did not want to distract from the visitors’ experiences. After 21-Tech, however, he learned the tablet was another tool for interacting with visitors to enhance the exhibits. For example, in a veterinarian exhibit, he helped visitors use the tablet camera to take photos of animals for adoption and then show the photos to other visitors, thus, increasing social interaction between groups. Provided Opportunities to Collaborate and Learn from Colleagues Facilitators reported that 21-Tech enabled collaboration and ongoing conversations among facilitators at their own site, facilitators at other sites, and other 21-Tech team members. The most prevalent collaborations were those among facilitators at the same institutions who compared notes on their experiences with visitors, apps, and exhibits. Thus, facilitators collaborated on solutions to problems, new techniques to use on the floor, new apps to try with visitors, and other exhibits at which to add facilitated experiences. These intra-institution Exposed Facilitators to New Technology Some facilitators said that 21-Tech was valuable to them because it exposed them to new technology. At the time of 21-Tech, most facilitators did not own a personal tablet, so the project allowed them to learn and become comfortable with the new technology. One early childhood educator, for example, commented that 21-Tech was especially helpful to her understanding of how to use technology in early childhood spaces in an appropriate manner and then model this use of technology for families. Deepened Facilitation Skills Facilitators noted that 21-Tech changed their facilitation practice for the better. Because they had access to more information via the tablet, some facilitators mentioned that they were more comfortable asking or being asked questions that they could not immediately answer. Thus, facilitators did not have to be “experts” but rather could function as co-learners with visitors. Experienced facilitators said that while saw little change in their facilitation practice, they regarded tablets as small, portable tools that were very useful to have in engaging visitors. Some of the exhibits we have are dated and there have been new discoveries that have come out….So, here [on the Internet] is more information on this topic and this [information] actually gets the visitors a little bit more interested too because now they can learn more about the topic if they are interested in it. There was a connection gap between what I knew and what I was trying to convey to them. So, this app became a bridge between what I knew—a better way to send information to a visitor and then how they can understand it. I want to get my own iPad because of how we use it on the floor…it empowers kids, especially if they can hold and move things. I appreciated getting the chance to be part of the project… I think the technology is really cool and that it is 1) an essential way to reach visitors and 2) the wave of the future. I think what it does is that if I do have the tablet device on me it lets me ask bigger questions in the sense that these are more thought-provoking questions…I think personally for me it helps me push what boundaries I can have with a visitor….So, now I can make you think even more and I can show you why. I think that what’s become much more obvious to me [over time] is that relationships between [facilitators] and visitors becomes much more of a collaboration …and [the] iPad being a catalyst of it. —Project Leader Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Facilitators, cont’d. conversations were both informal and formal. Informal interactions might happen when facilitators passed each other in the hall and might comment about what they had just tried out. Formal conversations took the form of meetings or focus groups to specifically discuss practices. While some sites held more structured and more frequent meetings than others, all sites had regular meetings with the intent of discussing 21-Tech implementation. Facilitators also benefitted from inter-site collaboration—via Skype sessions—with facilitators at different partner sites. Both inter- and intra-site communications allowed facilitators the opportunity to hear about other interaction scenarios and solutions to problems that they themselves might not have encountered. It promoted group problem-solving, which made the 21-Tech project more efficient and more valuable to the facilitators. Interviews with team leaders showed that in some organizations (particularly CMH), this group problem-solving model has the potential to be applied to future projects and initiatives well beyond 21-Tech. Empowered Facilitators Finally, ongoing conversations between facilitators and project leaders proved invaluable to the success of the project. Ultimately, it was the facilitators who implemented 21-Tech in the exhibits with visitors and who had first-hand knowledge of what was working and what could be improved. They were also important reviewers of apps and helped to link them to exhibits. Thus, involvement in 21-Tech also helped empower facilitators. Not only were their opinions being heard, but across sites, they also were given the power to research new apps and vet them. In some cases, facilitators took it upon themselves to start projects related to 21-Tech. For instance, one facilitator began an initiative to find apps that would be helpful for children with autism. At OMSI, a facilitator used an iPad to show a time lapse video of a 3-D printer working so that visitors could understand the entire process necessary to print. We were encouraged to bring the tablets with us anytime we were here. We’d bring them with us and sort of research as we are going through our day or practice what we’ve already established. And so, sometimes there would be a streak of genius like “Wow, that’s really great!” and you pass by and say to somebody “Do you think this would work?” or “How you do this?” Apart from getting to participate in the project like 21-Tech, it was really valuable to interact with the audience that I had. I also observed how technology can play a vital role in enhancing one’s interest in STEM activities. …when we did the Skype exchange…last month it was amazing….I think just communicating and knowing that we had each other and that we were going through the same thing, that was very helpful for me. [Another facilitator] and I would go over to an exhibit and we’d spend time there and we'd use the app in different ways…. It’s a longer collaboration and I think…more gets generated because we know that we have this group that we can go talk to. We know we have these other people…that have the same goals as opposed to just learning the exhibit and planning a way to tell people about it. It’s a deeper collaboration. [The most valuable thing about participating in 21-Tech was] Collaborating with the rest of the 21-Tech team to find new, innovative ways to expand upon the 21-Tech project. At CMH, a 21-Tech whiteboard was located where facilitators took breaks, thus encouraging facilitators to think about and discuss the project. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Visitors Visitors indicated that interacting with a facilitator who had a tablet helped them better understand the exhibits. The majority (74%) of visitors surveyed agreed strongly with the statement “Having the staff member with the tablet helped us better understand this exhibit.” The exhibit was fun by itself when I came here before, but now with the iPad I could see how the lines moved and what happens when [the table] shakes. —Child This exhibit and the iPad gave us a much more in-depth perspective about these things that we wouldn't have gotten through a less interactive exhibit. —Adult I appreciate the work the museum does to try to make some of the exhibits more substantial. I wish it was more prevalent though, because some of these kids are just here to wreck things. —Adult The iPad gave her a way to see, touch, and learn. And it was great to see her so happy at [the] end. We took so many pictures! —Adult It helped me understand the different proportions of plastic. It was a good visual tool. —Adult People need more time with the iPads [than we had with the facilitator]. If people could rent iPads for the day, like add another $1, that would be better. —Child The tablet helped a lot. We got to do it ourselves with Color Mixer. —Child Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Visitors, cont’d. • Observational data also showed that visitors were intellectually engaged during their interactions. Specifically, we found that in 62% of interactions, at least one visitor in the group asked a question, and of questions asked, nearly half (47%) were asked by children. This suggests that visitors engaged with the ideas and concepts explored in these interactions. • In addition to asking questions, visitors were observed making connections during interactions—both between exhibits and apps they were using and to their previous knowledge or experience. In 59% of interactions, at least one visitor in the group made an explicit connection between the exhibit and the app. Children were more likely to make these connections; 29 children made this type of connection versus 11 adults. • In 33% of interactions, at least one visitor in the group made a connection to his or her previous experiences. Examples of connections made by visitors included connecting plastics to garbage seen on a beach, connecting white blood cells to personal experience with cancer, and connecting building techniques to withstand earthquakes to a trip to Japan and the buildings there. • Visitors interviews following observations of their interactions with facilitators using tablets provided further insight into the ways visitors believed these interactions helped them at the exhibits. • The most common reason provided was that the visual nature of the tablet apps helped them understand the exhibits better. For example, one parent said that the Nice Molecules apps allowed her child to see a molecule in 3-D and in motion, which made it easier for her to make the model of it than if she had only seen a printed image. • In the case of the World Record Paper Planes app (paired with CMH’s Paper Airplane Launch Cage), parents noted that their children learned more because they were able to see different ways to build planes in a step-by-step process. In yet another example, a parent explained that when the facilitator showed his child photos of bridges, it connected the arch they were trying to build at SCI with real bridges. Visitors at OMSI talked about how the iPad Proscope was easier to use and the image easier to see than that of a microscope. Field Notes The eight-year-old boy was very interested in the different states of matter represented in the Phyzios application. He asked a question, in various ways, on “What different forms of matter would mix better with water?” The ten-year-old boy also enjoyed this application and asked “Is the reason this looks like this because the mixture is more of a liquid than a solid?” The mom continued to point out parts of the exhibit that corresponded to the molecules app, including helping the children count out pieces and pointing to signs detailing the types of molecules. The seven-year-old girl understood that the molecule pieces could be used to create the molecules on the iPad. They read exhibit signs out loud to label each piece on both the iPad and in person. The mother discussed with the facilitator how the cells on the iPad and what was on the microscope were different. The child noticed that there was also a difference between the microscope and iPad. The four-year-old boy’s question compared the colors on the wall and the colors on the tablet: “None of the colors [from the lights] can make those colors [on the app], right?” The seven-year-old girl’s [comment] showed that she probably already had experience mixing green: [After seeing the facilitator use Color Mixer to turn colors up and down, she said:] “That means we can make green!” She adjusted the colors and made yellow by combining red and green. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Visitors, cont’d. • Another benefit of facilitation with tablets mentioned by caregivers was that children could interact with the tablet on their own. These caregivers noted that they did not have to help their children use the app and felt positively about being able to give their children that level of independence in their explorations. • Visitors also said that the tactile experience of the tablet helped their family understand the exhibit better. These visitors explained that their children enjoyed the experience of being able to move objects on the screen. • Sometimes, apps that quantified a phenomenon helped visitors understand it better. For instance, the Vibrometer app used at NYSCI’s Bounce Back exhibit helped visitors more readily see how much energy the floor surface absorbed. The iSeismometer app being used with the When the Earth Shakes exhibit at SCI. This app augmented a purely physical experience with mathematical data and a visual display, enabling a richer conversation. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Visitors, cont’d. It showed stuff you couldn’t see otherwise. —Adult The tablet quantified it. Instead of [seeing that] one [floor surface] absorbed more or less, they saw a number. —Adult There are different ways to make a plane, not just one way. It was nice to be able to make something more difficult. —Adult It was easier to see the process of making the molecule visually on the iPad rather than the wall. —Adult We were able to move the iPad's image around and really magnify it. The microscope is much harder to use. —Adult It’s easier even for parents, [who] might not know the background [of the exhibit]. I’m a teacher. It can be overwhelming, but it’s useful to have a person around to help explain the exhibit. —Adult A book can just tell you about properties of matter, and the iPad shows you how they are different. —Adult I personally don't know anything about technology, so I thought it was very neat. I think it was great because it was very visual and she could easily use the program to move the pictures around. —Adult Seeing the tablets used at a science museum encourages them to try different science-related apps. —Adult In this video, a girl builds a model of an acetic acid molecule using the visual model on the tablet. Notice how the facilitator refers to both the image on the tablet and the exhibit labels when discussing molecules with the visitor. Her mother sits to the side, supervising a younger sibling. To play, click http://youtu.be/P9mwxsglpuk. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations Evaluation results also indicated that 21-Tech made important impacts at the organizational level for project partners. This section describes overall organizational impacts and provides a summary of the impacts at each partner organization. Strengthened Facilitation Team Skills In order to perform facilitation with tablets in an effective, natural way, facilitators had to become knowledgeable about the concepts behind the exhibits, comfortable with the technology, familiar with a variety of ways to introduce the tablet and interact with visitors, and even more cognizant of visitors’ interests and how to address their questions. This all resulted in facilitators “upping their game” regarding facilitation practice. Increased Understanding of Ways to Integrate Tablet-based Facilitation into a Range of Visitor Experiences Once partner sites learned how to effectively integrate tablets into exhibit experiences, they began to experiment with expanding use of tablets into other areas such as learning labs, programs, and even camps. For example, one CMH facilitator started her own initiative to use the iPads with special-needs children. She found an app to use with children with autism that displays faces expressing emotions and then asks the user to identify the one face with the different emotion (i.e., it would display three sad faces and one happy). At SCI, Counselors-in-Training used iPads to capture video of the camp experience. OMSI used iPads in the Maker Center of their Technology Lab to augment the 3-D printer experience. They recorded video of an object being made by the 3-D printer and then played the video later on an iPad by the 3-D printer, which was working on another object. The video allowed control the printing process and see the entire process faster than in real time. Partners thus deepened their awareness and understanding of the potential that tablets have and helped them reflect further about visitors’ needs and how tablets might help the organization meet those needs. Increased Commitment to Mobile Technology Partner sites generally became more committed to using mobile technology as part of facilitation following their successful experiences with 21-Tech. For instance, OMSI leaders said that not only was the Education Department planning to continue using the 21-Tech apps, but that the Outreach group expressed interest in them. OMSI’s Membership Services, Program Sales, Outdoor School, and Exhibit Developers also plan to meet with project leaders to explore adapting 21-Tech to their departments. At CMH project leaders plan to experiment using tablets in non-facilitated experiences. [The apps we use now are] going to continue to be part of our program as we build. I mean, we’re just going to think of them as another resource as we continue to develop new activities, new lab programs….I know other [departments], like our outreach department, they look over at the iPad very fondly, like, “Oh gosh, we need to get some of those.” We went around to the different exhibits and we thought “Okay, this would be really cool if we had something to add on to explain it.”…And so, we would search different phenomena that goes with that exhibit and see if there was any app that could explain [it]. In the beginning of the project, myself and [another facilitator]….were probably two of the most against using it [the tablet]….So it’s funny that we shortly became the ones who were showing everybody how great it is using it with visitors…because we didn’t quite understand how versatile they [tablets] were either. But, you know I think converting us two first was really good….Then, we were able to effectively show others like, “Come on, if we get into it you know anyone can.” Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. Provided Opportunities for Collaboration and Flattened Organizational Hierarchy The design of the 21-Tech project involved significant input from facilitators as part of developing, testing, and refining tablet-based facilitation. This structure empowered facilitators, man of whom shared that they now felt they had a voice in the project. Facilitators at all partner sites were also free to choose when and where to use tablets with visitors. This empowerment is particularly noteworthy because facilitators tended to be the least senior staff in the organization (younger and less experienced staff often new to the organization, part-time employees, volunteers, or interns); 21-Tech, in a sense, helped them gain more equal footing. Facilitators felt positive and proud to be able to contribute in substantive ways to the project and to have an impact on the organization. Encouraged a Reflective Community of Practice The evolving nature of the project, which required testing and refining apps and facilitation strategies, necessitated that partners engage in reflective practice. All partner sites said that they benefitted from the community of learners that sprung up around the 21-Tech project. This community encompassed the entire 21-Tech team, from the project leaders to the facilitators. Project leaders spent time exploring and experi-menting with the technology, learning from each other as well as from facilitators. Figure 16. Models of communication prior to and after 21-Tech Managers Prior to 21-Tech implementation Facilitators Visitors Visitors After 21-Tech implementation Facilitators Managers The graphic above depicts how 21-Tech flattened traditional hierarchical communication models between managers and facilitators. 21-Tech also improved communication between visitors and facilitators, who were able to become more responsive to visitors though the use of tablets. As a result, managers, facilitators, and visitors listened to each other more and often become co-learners. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. Leaders noted that they all learned from each other and found the involvement of very different institutions helpful. For example, a SCI project leader discussed how useful it was to learn how the three other sites (which they called “grand-daddies”) implemented 21-Tech, knowledge that allowed them to more easily adapt it to SCI. This spirit of experimentation set the tone of the project; no one person was seen to have all the answers. The facilitators themselves formed a community of practice around 21-Tech facilitation, which, with some institutional support and encouragement, became self-sustaining; facilitators were excited to continue to refine and discuss 21-Tech with one another, which led to a sense of ownership of the project. There were also some indications from project leaders that the cohesiveness and collaboration developed among staff will continue to benefit their organizations. Strengthened Ties Among Institutions Project leaders at all four sites also felt that involvement in 21-Tech had strengthened ties among the four partner organizations. Some commented that this was the most valuable aspect of the project for them. Partner leaders shared ideas, tried different things in the spirit of experimentation and collaboration, and together built knowledge and capacity to use tablets in facilitation. For instance, NYSCI pushed the project technologically by using both iPads and Nexus tablets and made sure that the team considered both iOS and Android operating systems in terms of apps. Another example is the way partners could read each other’s blog posts on 21-Tech.org to understand how similar apps could be used in different ways. So both from a personal standpoint, it was nice to have that chance to sort of reflect on my own practice and look at the tools I use in my practice as an educator. It also allowed me to look at my connection with the team I have here with facilitators and educators and how we interact here. Then also…connecting with other museums and the educators and program developers there. The conversation continues on the floor…spontaneously on their [the facilitators’] own or they’ll talk in the hallway…It’s like when you do something really fun, and you just want to share it with someone. So, when they would have a really great interaction, they just wanted to tell someone. Instead of just having our own experiences, by having it be part of 21-Tech with this larger community, we had a body of experiences that raises my confidence as I think about researching and developing new ways to engage the visitor or the next places to take personal mobile technology. [21-Tech] reinforced relationships that we had with institutions and…started relationships that I didn’t have…but they’ve given me ideas in ways that we can take the direction of our work. A close-up of the 21-Tech board at CMH. A different app was featured each week, with a description and suggestions of which exhibits to use it with. There was also a place for facilitators to post comments about the app, which served as a mechanism to provide feedback and share perspectives. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. While the organizational impacts described in previous pages held true across partner sites, contextual factors influenced both process and outcomes. In the following pages, we examine each organization in more detail to illustrate some unique ways in which 21-Tech played out at each partner site. What we’re contemplating in several upcoming exhibits is a direct result…of a very positive experience with 21-Tech—what we’ve been able to see visitors are capable of doing with these tools and what we’re able to do with these tools. It’s a struggle to learn 21-Tech; it’s a struggle to get better at the practice as a facilitator. They always know that [the project manager is]…encouraging them, out there with them, and always there for a conversation about how to make our collective practice better. It was amazing ideas that they [the facilitators] were coming up with...We’ve done a professional development with teachers last weekend where we showed them how to use iPads in the classroom. We’re using them in science workshops….We’re using them in parent workshops to educate parents about what else you can do with a tablet besides showing [videos]. Children’s Museum of Houston CMH was lead partner on the 21-Tech project. As lead partner, CMH had the strongest institutional commitment to the project. Furthermore, the team leaders—one of whom was a senior executive—had the assistance of a dedicated project manager. CMH implemented 21-Tech as an enhancement to their established gallery facilitation. CMH had 82% of their floor staff of 55 trained for 21-Tech. Training was approximately 6–8 hours over 4–5 weeks. CMH was the only site of the four partners with a dedicated 21-Tech project manager. After starting with a project manager who had a more rigid, traditional approach to managing facilitators (and where facilitation staff felt somewhat “forced” to participate in 21-Tech), a new project manager with a more collaborative approach was hired. This manager became a “hub” for facilitators and project leaders. Her coaching and mentoring skills and accessibility (being open to discussions and physically located near facilitators) allowed facilitators to form a community of practice and also allowed for communication to flow more effectively between leaders and facilitators. The project manager also created CMH’s powerful training and feedback model based on weekly facilitator focus groups. This model, which supported facilitator empowerment, was viewed by project leaders as a new professional development and leadership model for CMH floor staff and educators. They believed that it was a practical demonstration of a concrete and systematic way to capitalize on the expertise of the on-the-floor facilitators. Another impact unique to CMH involved the relationship of 21-Tech to CMH’s strategic plan. CMH had an existing technology initiative that was part of a strategic plan to incorporate technology more effectively from the perspective of its relevance to visitors, and the 21-Tech project shifted their thinking about incorporating technology as an institution. For instance, in addition to incorporating apps and technology at exhibits, CMH’s Outreach, Library, and Science Workshop also began using tablets in their work. iPads were also used during school tours and teacher professional development. As a result, 21-Tech moved the entire organization along its strategic technology plan. CMH facilitators used tablets in popular exhibits, such as the Vet Clinic in the Kidtropolis area, to supplement the role play already taking place. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. New York Hall of Science NYSCI is a large science center with a staff of 67 facilitators providing visitor experiences at their exhibits. 21-Tech implementation at NYSCI waspart of a larger tech initiative (which includes app development for specific exhibits) being undertaken at the institution. NYSCI had 6% for their floor staff trained in 21-Tech facilitation. Training lasted approximately 6 hours over 2 weeks. NYSCI was already involved in an initiative to develop exhibit-related apps and they already had an institutional commitment to experimenting with apps and PMTs. For NSYCI, 21-Tech became another important effort to actively explore how mobile technology could be used. Explainers and the leadership team involved with NYSCI were strongly impacted by their involvement, particularly in terms of strengthened facilitation skills and their ability to customize the visitor experience. Furthermore, facilitators came to see apps not just as games, but as useful tools for learning about science content. As a result of facilitators’ experiences in 21-Tech, the organization then began introducing tablets and related apps as part of training new facilitators on exhibits. Project leaders reported that this had been a useful addition to training. Additionally, the process of involving facilitators in the 21-Tech development process helped illuminate the facilitators’ training needs more broadly. More specifically, project leadership shared that their experience in 21-Tech and the ways the program supported facilitators’ professional development helped them reflect more deeply about the kind of support and training facilitators need when incorporating new technology into facilitation. While having two initiatives focused on mobile technology was helpful in that 21-Tech helped leadership think about PMT-based facilitation, it meant fewer staff could be involved. 21-Tech, while not taking the organization in different directions, did complement their existing initiatives and deepened their understanding of incorporating mobile technology in facilitation. One challenge NYSCI identified was the turnover of the project leader at their site. The need to “get current” on the project and the time it takes to transition (e.g., a new team member fitting the project into her current responsibilities, getting up to speed on a complex project, facilitators becoming used to a new project leader, etc.) factored into the process. Overall, staff members involved with 21-Tech found it a valuable professional development effort. Project leaders reported that 21-Tech enhanced the technology initiatives already underway. I think that was probably the biggest impact for me just having [tablets]...in training the Explainers, who will later on use with visitors. I think as an organization we might be doing a lot of app development…So, it [21-Tech] was kind of an add-on, because a lot of that kind of precursor work was happening while 21-Tech was starting. So, it didn’t take us in any different directions but…it did enhance the direction we were already going in. NYSCI facilitators used tablets at exhibits such as those relating to light and pitch. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. Oregon Museum of Science and Industry OMSI is a large science center with 100 volunteer facilitators. At OMSI, 21-Tech was implemented in various ways. 21-Tech facilitation with tablets was generally used in demonstrations and to engage visitors who were waiting in line for other activities. Senior educators also added 21-Tech facilitation to their labs and classes and even used tablets to show visitors what research they were doing behind the scenes in their labs. OMSI trained 25% of their floor staff in 21-Tech and also involved senior educators and youth volunteers from their Rising Star program. The 21-Tech training at OMSI lasted approximately 5 hours over 6–8 weeks. Initially, OMSI experienced some resistance from facilitators—particularly senior educators—who were skeptical that tablets could add anything positive to their already robust, hands-on facilitated visitor experiences. One of the project leaders, however, acted as a strong advocate to convince others of the value of tablet technology. These efforts helped address concerns and better engage staff. Interestingly, once the two most skeptical facilitators were aboard, they became two of the project’s top advocates and helped other staff see the value of tablets as facilitation tools. the galleries. The need for more reliable access spurred the the project team leader to advocate for strengthening the Wi-Fi signal throughout the science center and making good Wi-Fi a priority. OMSI experienced project team leader turnover, as did nearly all the other partners, but when one of the 21-Tech leaders left the project, one of the two initially skeptical facilitators was able to take his place. This substitution of a person committed to 21-Tech helped make for a smoother transition than bringing someone completely new into the project. 21-Tech also encouraged more sharing and communication among staff and helped to connect different labs and areas in an institution that otherwise operates fairly independently. For example, staff from two different areas found that an app related to earthquakes could be used to talk about forces in both the earth science and physics halls. Project leaders said this type of connection between areas of OMSI could not have happened without 21-Tech. OMSI project leaders also found that the tablets helped senior educators better communicate to visitors the educators’ personal lab research and even allowed them to demonstrate how scientific research is done. For example, a life science lab educator used iPads to show home-school students how to research and filter content on their own. The 21-Tech project also was an influence on OMSI improving its Wi-Fi infrastructure. Initially, staff struggled with implementing 21-Tech because of the spotty Wi-Fi access in [During the 21-Tech project] We were simultaneously working on a…bilingual Spanish-English cell phone campaign and QR codes, and some of these other aspects of how we can meet people where they are through the technology they use, without things being quite so expensive as building big exhibits from the floor…So, some of this [21-Tech] work was simultaneous with that thinking and was really back and forth and really important to us. At OMSI, tablets were used in demos to enhance the experience. It allowed facilitators to answer visitor questions by accessing more information quickly. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. program for middle-school teenagers, so they could learn about mobile technology as well as how to interact with visitors and even develop apps. Sciencenter SCI is a small science center that caters to families with children younger than 8. Its emphasis on hands-on activities, coupled with budget constraints, meant that SCI, of all the partner sites, had the least amount of technology on the floor. It also had the smallest staff of facilitators (23 volunteers/ interns), one that experiences frequent turnover. Prior to 21-Tech, SCI provided facilitated interactions only around tabletop activities. Implementing 21-Tech at SCI, thus, involved a major change in their interaction model. SCI had 25% of their facilitators involved in the 21-Tech project. Their training lasted approximately 4–5 hours over 4 weeks. Implementing 21-Tech at SCI meant that the project team had to introduce tablet-enabled facilitation and develop a new interaction model for exhibits. Involvement in 21-Tech resulted in creating more in-depth educational visitor experiences directly related to exhibits. Involvement in the project also helped SCI more deeply integrate volunteer/intern facilitators into projects. Due to the complexity of 21-Tech and the need to gather facilitators feedbackabout the floor experience, project team leaders had to find ways to get their voices into the process. 21-Tech also pushed the organization to reconsider its perspective about the use of mobile technology. The use of free or inexpensive off-the-shelf apps demonstrated that SCI could successfully integrate technology in a cost-conscious way. It also helped provide strong evidence that such technology could enhance physical exhibits instead of competing with them. Project leaders reported that their involvement in 21-Tech also gave them experience and knowledge that the entire organization could tap into. For example, the organization’s Executive Director saw the project leadership team having gained valuable experience in 21-Tech project and so saw them as experts when questions arose about the use of apps at exhibits. 21-Tech also impacted other SCI projects and departments. In addition to 21-Tech facilitation, iPads were also used by the Education Department for activities such as capturing video for camps and facilitating citizen science projects. In fact, project leaders reported that SCI’s Director of Programs and Exhibitions also became more excited about using different technologies as part of exhibits. SCI project leaders indicated that they planned to continue using 21-Tech with their volunteers. The organization also had plans to adapt it to their Counselors-in-Training I think 21-Tech has sort of helped us change the conversation from…computer kiosk kind of exhibits, built around a screen, to thinking about ways we can be more creative in using technology to enhance the visitor experience of the physical experience. Because we have the iPads, everyone is finding different uses for them now…it becomes an institutional resource at a place our size. 21-Tech facilitators at SCI often interacted with very young children at their exhibits. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. Examining the ways in which 21-Tech was implemented at the different partner sites illuminated the myriad contextual variables at play. Several factors influenced both implementation and levels of organizational impact. Size of Organization Large organizations had the benefit of more people to support 21-Tech. These organizations were specialized enough that at least some 21-Tech team members could devote significant time to the project. On the other hand, at SCI, a small institution, 21-Tech team members had competing priorities for their time. Conversely, larger institutions were challenged to disseminate the 21-Tech project within their organizations. It was much easier for SCI staff not involved with 21-Tech to hear about the project than at a larger site such as NYSCI or OMSI, where more formal dissemination is required. Number and Employment Status of Facilitators Organizations with more facilitators had more people to draw on for involvement in the 21-Tech project. This also usually implied a more formalized training program into which 21-Tech facilitation training could more easily be added. Additionally, having a larger number of 21-Tech training facilitators (the case at, for example, CMH) meant that facilitators had a larger peer group to collaborate with on the project. Organizations of mainly volunteer facilitators faced some challenges. Volunteer staff, in general, had higher turnover rates, less time on the floor, and more irregular hours than paid facilitators. As a result, it could be more difficult to train volunteer facilitators and put their skills into practice consistently. Additionally, volunteers are typically not involved in project development. Therefore, it was necessary to develop new processes to elicit feedback from volunteer facilitators. Project Leader Turnover Some partner sites were challenged when project team leaders left during the 21-Tech project. While transition plans were always in place, getting new members up to speed still required time. This was especially the case when few senior staff members were involved in the project leadership team. Consistent project leadership made the impact of 21-Tech stronger at some institutions than others. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Partner Organizations, cont’d. One of the important findings of the 21-Tech project was that tablet-based facilitation could work in very different types of institutions and was adaptable to different organizational contexts. Program qualities that helped 21-Tech succeed in different settings included: Garibay Group | 21-Tech | Summative Evaluation | February 2014

Results: Dissemination Garibay Group | 21-Tech | Summative Evaluation | February 2014

Dissemination Efforts 21-Tech held four workshops, two webinars and one discussion forum toshare information about the project with museum professionals. Three of these sessions were evaluated as part of the summative evaluation—the ACM pre-conference workshop, the ASTC pre-conference workshop, and the “21-Tech Overview” webinar. • Over the course of the project, 21-Tech held the following: • In partnership with ASTC, a conference session in October 2012 at the ASTC Annual Conference in Columbus, OH. One hundred participants were seated, with additional participants standing, to hear an overview of the project. • In partnership with ACM, a half-day pre-conference workshop in April 2013 at the ACM Annual Conference in Pittsburg, PA. This workshop was held at Carnegie Science Center, and had 29 participants from science centers as well as children’s museums. • In partnership with ASTC, a full-day pre-conference session in October 2013 at the ASTC Annual Conference in Albuquerque, NM. Twelve members from the partner team hosted an eight-hour session at Explora! Science Center with 27 national and international participants from science centers and children’s museums. • A WebWisedemonstration/showcase at the Annual IMLS Conference in February 2013 about adaptation of third-party apps with hands-on exhibits. At least two hundred museum professionals had the opportunity to view the demonstrations within the two-hour session. • A one-hour “21-Tech Overview” synchronous webinar hosted by ASTC in September 2013. This webinar had 118 registrants and at least 87 real-time attendees. Since this webinar was recorded, it serves as an asynchronous session that can be accessed atASTC’s Vimeopage. • A one-hour synchronous facilitator webinar hosted by ASTC in November 2013. This webinar had18 attendees and 6 staff members online. This webinar was also recorded and archived and serves as another asynchronous session. It can be accessed via ASTC’s Vimeopage. • A two-week long asynchronous discussion forum hosted by ASTC in November 2013. This forum remained the most active discussion on ASTC’s community in November, with a total of 51 members and 56 posts. This forum will be archived on the ASTC site as an open resource for other museum professionals. A 21-Tech team member helps an ASTC pre-conference workshop participant try an iPad app with an exhibit. Garibay Group | 21-Tech | Summative Evaluation | February 2014

21-Tech Workshop Outcomes The majority of workshop survey respondents (86%) indicated that the workshop met or exceeded their expectations. I was surprised to learn just how much of the 21-Tech methodology is already in place at my Museum. I think the 21-Tech website will be a valuable resource for finding new apps and hardware that we can integrate into our existing program. When I’ve worked with many of our (particularly younger) volunteer facilitators, I’ve had a hard time with them basically ignoring the exhibit/activity in favor of the iPad. The way you articulated the focus on the exhibit will help me convey this to facilitators and hopefully keep the focus more on the hands-on! I liked the approach for interacting with the visitors and the training the facilitators went through. It will be useful, not only for when we begin to incorporate this type of mobile technology, but [also] for the exhibit enhancements we already use as well. [I learned] That there can be value-added uses of technology, but [that] there is a lot more to it than just handing out iPads. Don’t be naive in terms of the level of investment to do it correctly and questions on sustainability need to be addressed. Garibay Group | 21-Tech | Summative Evaluation | February 2014

21-Tech Workshop Response, cont’d. The workshops successfully increased participants’ understanding about tablets as facilitation tools. Respondents indicated that the most significant increases in their understanding of the role tablets could play came in enhancing visitor learning and incorporating tablets into facilitation at exhibits. Ratings for increases in understanding of how to choose apps and match them with exhibit components were more spread across the scale, with the majority indicating that they experienced moderate increases in their understanding. Table 7. Workshop participants’ understanding ratings of the question, “How much did the workshop increase each of the following?” The biggest takeaway was using apps as a tool to enhance exhibits. That changes the way I look at which apps to download and use. I liked the idea that the tablet is an extension or another tool, rather than the [end]-all and be-all. [It was useful for me to learn the] importance of incorporation of technology and showing visitors that kids (and adults) can use them as an educational tools in many ways. I think the importance of using the tablet to support and scaffold from exhibits is great! it would be great to go even further exploring the synergistic opportunities between the interaction of educator/facilitators and exhibition. n = 35 Garibay Group | 21-Tech | Summative Evaluation | February 2014

21-Tech Workshop Response, cont’d. Participants found the workshops professionally valuable and also reported increased motivation to use tablets at their institutions. Nearly all respondents said that the workshop was at least slightly valuable to them professionally, with almost half (46%) rating it as “very valuable.”Furthermore, 89% said the workshop “moderately” or “significantly” increased their motivation to incorporate tablets as part of visitor facilitation. Garibay Group | 21-Tech | Summative Evaluation | February 2014

21-Tech Webinar Response The Webinar also saw solid expectation and satisfaction ratings. Most (84%) participants surveyed said that the 21-Tech webinar met or exceeded their expectations, with 87% indicating they were satisfied or very satisfied with the webinar. We are an emerging museum exploring technology options for our exhibit program. The webinar provided some very useful perspectives on the subject. It was exactly what I expected. I wish it was a longer session with more chance for examples. I work for a museum that is on more of the low-tech spectrum. It was great to see how well integrated the staff was with tablets and HOW they were used to engage the visitors. The applications discussed for tablet implementation seemed useful and we may deploy a couple of them, but [it was] not mind-blowing. It was very useful to see the use case examples —what really works. The information was much more practical than I had expected, and it was well backed up by additional information for after the webinar. We were hoping to hear more about how to engage visitors on their own devices that they tend to have with them anyway. Garibay Group | 21-Tech | Summative Evaluation | February 2014

Impact on Museum Professionals: Webinar We asked questions about increased understanding in four specific webinar foci. Respondents reported moderate increases in all categories. Table 8. Respondents’ ratings of the statement, “How much did the webinar increase each of the following?” n = 31, except n = 30 for “Your knowledge of examples…” Garibay Group | 21-Tech | Summative Evaluation | February 2014

21-Tech Summative Evaluation Report