Equalia – An Evaluation System in the Application Context of Technology-Enhanced Learning

1. Equalia – An Evaluation System in the Application Context of Technology-Enhanced Learning Eva-Catherine Hillemann Alexander Nussbaumer Christina M. Steiner Dietrich Albert EU-Day @ I-Know 2013 Synergy Workshop, Wednesday, Sept. 4

2. The Project: CULTURA • CULTivatingUnderstanding andResearch throughAdaptivity • Project type: Small or medium-scalefocusedresearchproject (STREP) • Funding Body: European Commission, FP7-ICT • Start date: February 2011 • Duration: 36 months

3. The Project: CULTURA • Vision and goal • CULTURA will provide innovative adaptive services and an interactive user environment which dynamically tailor the investigation, comprehension and enrichment of digital humanities artefacts. • Objectives • CULTURA will deliver innovative personalization which takes into account • individual user intent, motivation and diversity of use • awareness of the activities and interests of the community to which the user belongs • in-depth analysis of the structure and features of digital humanities artifacts and collections  Detailed understandings of the user, their community and content

4. The Project: CULTURA TU Graz Contribution • Evaluation • Methodology and plan for sound evaluation of the whole CULTURA system and its functionalities • Conduction of empirical evaluations with user communities • Design and implementation of an evaluation tool

5. Evaluation I • Evaluation is a process that critically examines a program • Involves collecting and analyzing information about program’s activities, characteristics, outcomes; • Purpose: making judgments about a program to improve effectiveness, and/or to inform programming decisions. Why evaluate? • To improve usability • To increase user acceptance • To enhance learning outcomes • To increase learning efficiency

6. Evaluation II • What do I want to do with the information? • Informing design • Formative evaluation • Assessing end product • Summative evaluation adapted from Norland (2004), Pancer& Westhues (1989), Rossi, Lippsey, Freeman (2004)

7. Evaluation III • How do I do evaluation? • Stages of an evaluation process • Planning • Purpose & Type of Evaluation • Stakeholders • Evaluation Questions • Evaluation Plan & Methodology • Carrying out • Applying instruments • Data Collection • Working with results • Data Analysis Method • Data Analysis • Report on Results Cook, 2002

8. Equalia – An Evaluation System I • Support evaluators in designing, carrying out, and analysing evaluation • …in different evaluation contexts • Reusability and interoperability for the usage • …using different evaluation modes • continuous and non-continuous, invasive and non-invasive

9. Equalia– An Evaluation System II Main features in order to support the evaluation process and allow for systematic evaluation: • Evaluation model • Used for conceptual definition of what is evaluated • Different evaluation modes • Continuous and non-continuous evaluation • Invasive and non-invasive evaluation • Evaluation reports • Different user interfaces • Participant (questionnaires) • Evaluator (management of evaluation model and reports) • Self-contained system

10. Conceptual Design & Approach I • Underlying evaluation model • What should be evaluated? • Evaluation qualities  Quality model • How should be evaluated? • Evaluation modes • Survey model • Authoring tool for creating evaluation model

11. Conceptual Design & Approach II • Evaluation modes

12. Conceptual Design & Approach III • Evaluation report • Triangulation of different datasources • AutomaticallycreatedbyEqualia • Based on underlyingevaluationmodel • Meanvaluesandaggregationsofrawdata • Generation ofdiagrams • Data export

13. Applying Equalia to TEL I: Example • SRL-Learning Environment “Information Search & Retrieval” • use skill-based visualizations to support self-regulated personalized learning • Goal-Setting Tool • Learning Tool • Reflection Tool

14. Applying Equalia to TEL II: Example • Evaluation Questions & resulting Evaluation Qualities • Technical perspective • How usable is the SRL learning environment? • Usability • Do learners consider the SRL learning environment acceptable and intend to use it? • User acceptance • Psycho-pedagogical perspective • Does the SRL learning environment support self-regulated learning? • Support of SRL

15. Applying Equalia to TEL III: Example • Mapping these qualities to individual items • E.g. Support of SRL • Sensors • SRL as an activity occurring during the system usage • Analysis of learning activities • Indicator for monitoring; frequency of visiting the reflection tool • Questionnaires • MSLQ1 • Assessment of the general tendency to use learning strategies [planning, monitoring, regulating] • …Questionnaires specifically created 1Pintrich, P.R., Smith, D.A., Garcia, T., & McKeachie W.J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). National Center for Research to Improve Postsecondary Teaching and Learning. Ann Arbor: University of Michigan.

17. ApplyingEqualiato TEL IV • User interface

18. Applying Equalia to TEL V • Evaluation Report

19. Conclusio • Advantages • Systematic & soundevaluation • Specificationof an evaluationmodel • Formal definitionof WHAT & HOW toevaluate • Quality Model and Survey Model • Reusabilityandinteroperabilityforusage in different evaluationcontexts • Different modesfordatacollection  Application in other contexts/projects

20. Thank You! CULTURA Project: http://cultura-project.eu/