A Lifecycle Approach to Evaluating MyArtSpace

1. A Lifecycle Approach to Evaluating MyArtSpace Giasemi Vavoula with Julia Meek Mike Sharples Peter Lonsdale Paul Rudman

2. Outline • What is MyArtSpace • What is Lifecycle • Using Lifecycle to evaluate MyArtSpace • Results: Mobile learning for schools in museums • Conclusions g.vavoula@gmail.com, g.vavoula@open.ac.uk

3. MyArtSpace – Situation of concern • School visits to museums • Visits often isolated from classroom work • “Learning is a cumulative process involving connections and reinforcement among the variety of learning experiences people encounter in their lives: at home, during schooling, and out in the community and workplace” (Dierking et al. 2003) • “making the links between school and museum learning explicit, genuine, and continuous affords real opportunities for school students to have enjoyable learning experiences in both settings.” (Griffin 2004) • Making the visit personal and relevant • Static displays, generic labels • Structuring enquiry learning. Children need specific support in: • planning appropriate investigations • managing investigations • interpreting results (de Jong et al., 2006)  Design linked pre, on, and post-museum visit learning experiences  Personalise interaction with exhibits  Design enquiry-led museum learning g.vavoula@gmail.com, g.vavoula@open.ac.uk

4. MyArtSpace – What it is • Service on mobile phones for enquiry-led museum learning • Aim to make school museum visits more engaging and educational • Learning through structured enquiry, exploration, connection • Combines • physical space (museum, classroom) • virtual space (online store and gallery) • personal space (mobile phones) • Museum test sites • Urbis (Manchester) • The D-Day Museum (Portsmouth) • The Study Gallery of Modern Art (Poole) • About 3000 children during 2006 g.vavoula@gmail.com, g.vavoula@open.ac.uk

5. MyArtSpace – How it works • In the classroom: • Discuss ‘big question’ to explore, by collecting evidence from the museum visit • At the museum: • Students are given Nokia 6680 multimedia mobile phones • Use phones to ‘collect’ exhibits by typing a two-letter code • Prompted to type their reason for collecting • Document their experience (take pictures, record sounds, write comments) • All exhibits and recordings are sent automatically to a personal web space • Back at school: • Personal website shows their notes, recordings, pictures, exhibits • They can view others’ collections, and items provided by the museum • They can create personal online galleries to show teachers and classmates, and to friends and family outside school g.vavoula@gmail.com, g.vavoula@open.ac.uk

6. Lifecycle – What it is • A methodology for the evaluation of Educational Technologies • Basic premises: • Start evaluation very early in the design process – and keep evaluating throughout • Undertake evaluation activities at key points addressing questions: • Who are the stakeholders and the users of this evaluation activity? • Why is thisactivity being undertaken (purpose/goals)? • What evaluation methods/resources will be used • When should the evaluation be undertaken? • How will the results be communicated? • Feed evaluation outcomes into following stages, or re-iterate previous stages • Lifecycle Evaluation Toolkit g.vavoula@gmail.com, g.vavoula@open.ac.uk

7. Lifecycle – What it is (2) • Shares concepts and methods with software engineering, educational and learning technology evaluation (User-Centred Systems Design, lifecycle approach to software engineering, etc.) • Emphasises the central role of Evaluation throughout learning system design, from system conception to final assessment. • Provides a framework for planning evaluation activities and using their outcomes. Meek (2006), Meek & Sharples (2001) Contact: n.meek@ntlworld.com g.vavoula@gmail.com, g.vavoula@open.ac.uk

8. Lifecycle for MyArtSpace - Stakeholders • The Sea: MyArtSpace design and development team • Culture Online (UK Dept. for Culture, Media and Sport): funding body • Teachers: users • Students: users • Local Education Authority reps: expert users / promoters • Museum curators / education experts: users / clients / test sites • Education/accessibility consultants: experts g.vavoula@gmail.com, g.vavoula@open.ac.uk

9. Lifecycle for MyArtSpace – Evaluation activities g.vavoula@gmail.com, g.vavoula@open.ac.uk

10. Lifecycle for MyArtSpace – Evaluation activities (2) Jan 05 Project Timeline Conception • Consultation workshop • Development team, funding body, evaluation team/mobile learning experts • Improvement of initial concept • Requirements workshop • Development team, funding body, teachers, LEA reps, evaluation team, museum reps, accessibility expert • Educational, user and accessibility requirements • Usability evaluations • Heuristic evaluations of mobile phone, student/teacher web interface, museum curators web interface; at critical points in interface design • Field studies • Full scale trial with KS2 students at Urbis; main focus: system performance, usability • Full scale trial with KS3 students at D-Day; main focus: issues from previous trial, educational experience, usability • Final full scale trial with KS3 students at D-Day; main focus: educational and user experience, system impact • Stakeholder interviews • After-pilot deployment interviews; focus: great expectations – fulfilled? • Teacher surveys • Questionnaires and semi-structured interviews with teachers; main focus: what works and what doesn’t, why, perceptions and attitudes, space for improvement May 05 Consultation Jun 05 Design Aug 05 Build Jan 06 Deploy Dec 06 Market g.vavoula@gmail.com, g.vavoula@open.ac.uk

11. Lifecycle for MyArtSpace – Use of evaluation output Usability evaluations Formulate concept Field Studies Teacher evaluations Stakeholder Consultation Requirements analysis MyArtSpace marketing MyArtSpace system MyArtSpace concept MyArtSpace designs Evaluation activity System development stage g.vavoula@gmail.com, g.vavoula@open.ac.uk

12. Lifecycle for MyArtSpace – A focused framework • Emergent evaluation framework for mobile learning systems and services g.vavoula@gmail.com, g.vavoula@open.ac.uk

13. Lifecycle for MyArtSpace – Preliminary results • Mobile learning for schools in museums: • Micro level (individual learning activities) • Students enthusiastic about creating their own content: taking dozens of pictures and recording lots of audio • Text comments not very popular – in recent trial, they override text comments and record audio instead • Meso level (the learning experience) • “Made me look at artwork more” • Students stop before exhibits and think – should I collect? Is this relevant? • Prompted to abstract main ideas of their galleries by e.g. asking to provide a good title • Durable visit output to take away • Macro level (the educational experience) • Connect museum visit to school learning (“didn’t really use museum visits for specific purposes (before)”, “give us an easier follow up lesson to make the value of the trip more worth while”) • Increase engagement g.vavoula@gmail.com, g.vavoula@open.ac.uk

14. Lifecycle for MyArtSpace – Preliminary results (2) • Example inter-level influences (micro-meso): • ‘Aggressive’ collecting: the more open and exploratory the learning task is at the meso level, the more overuse is made of collection activities at the micro level… • … which makes harder the post-visit manipulation of collections at the micro level, which in turn hinders the learning experience at the meso level • learning task needs to be well designed and appropriately constrained • pre, on and post museum experiences need to be allocated enough time • No explicit link between pictures and comments at micro level results in difficulty in interpreting pictures post-museum at the meso level • Functionality needed: annotate picture with audio/text comment g.vavoula@gmail.com, g.vavoula@open.ac.uk

15. Conclusions • MyArtSpace: • Encourages students to stop and think about each exhibit • Engaging, fun • Consolidates formal and semi-formal learning experiences • MyArtSpace is being re-branded as OOKL for commercial release • Lifecycle: • Focus on both usability and pedagogy early in the design • Helped draw a full picture of the use of new technology to be understood, assessed and improved • Involved stakeholders in evaluation activities early on, making it everyone’s business to provide input to evaluation activities and make use of the outcomes! g.vavoula@gmail.com, g.vavoula@open.ac.uk

16. Partners - contacts • Culture Online (Department of Culture, Media and Sport) • http://www.cultureonline.gov.uk/ • The SEA – software development • http://the-sea.com • University of Birmingham (CETADL) • University of Nottingham (LSRI) • Urbis, Manchester • Study Gallery, Poole • D-Day Museum, Portsmouth • Lifecycle Ltd • Contact: Julia Meek (n.meek@ntlworld.com) • Partners: Julia Meek, Peter Lonsdale, Paul Rudman, Mike Sharples, Giasemi Vavoula (MAS evaluation team leader), <other two> g.vavoula@gmail.com, g.vavoula@open.ac.uk