Evaluating learning designs through the formal representation of learning patterns

1. Evaluating learning designs through the formal representation of learning patterns • Diana Laurillard and Dejan Ljubojevic • London Knowledge LabInstitute of Education, London

2. A Learning Design Support Environment (LDSE) Oxford Liz Masterman (CoPI) Marion Manton (CoPI) Birkbeck/LKL George Magooulas (CoPI) Patricia Charlton IOE/LKL Brock Craft (Technical) Sarah Gelcich (Admin) Diana Laurillard (PI) DejanLjubojevic (RF) Javier Calzada-Prado (Intern) LondonMet Tom Boyle (CoPI) LSE Steve Ryan (CoPI) Ed Whitley RoserPujadas(Studentship) RVC Kim Whittlestone (CoPI) Stephen May Carrie Driver (Studentship)

3. OUTLINE Evaluating learning designs Teachers are interested in learning design and the use of technology but lack the tools they need  LDSE Planning, advice, exemplars – could it also evaluate a design? • Design principles in the literature • Learning design patterns – to capture pedagogic principles? • Representing learning theories as an evaluative framework • Evaluating the pedagogy in a learning design • Testing the evaluative capability of the framework

4. Design principles in the literature Good feedback practice: 1. helps clarify what good performance is - goals, criteria, expected standards 2. facilitates the development of self-assessment (reflection) in learning 3. delivers high quality information to students about their learning 4. encourages teacher and peer dialogue around learning 5. encourages positive motivational beliefs and self-esteem 6. provides opportunities to close the gap between current and desired performance 7. provides information to teachers that can be used to help shape the teaching. (Nicol& MacFarlane-Dick, 2006)

5. Design principles in the literature 1. Encourage contacts between students and faculty 2. Develop reciprocity and cooperation among students 3. Use active learning techniques 4. Give prompt feedback 5. Emphasize time on task 6. Communicate high expectations 7. Respect diverse talents and ways of learning. (Chickering and Gamson, 1991) Where is the pedagogy that aids learning design? What active learning techniques?What kind of feedback? Can pedagogical patterns assist with designing and evaluating the quality of a learning design?

6. Learning design patterns • The LDSE( learning design support environment) aims to evaluate the pedagogy in a learning design in terms of theory • We look for pattern templates that will help to evaluate pedagogy in a design • A learning design pattern as a learning activity sequence designed to lead to a specific learning outcome

7. Comparing pedagogy pattern templates Where is the effective pedagogy captured?

8. Comparing pedagogy pattern templates Pattern for understanding authentic practice Pattern for helping whole class benefit from individual experiences Needs further detailed analysis to support learning design Categories do not sufficiently discriminate the form of pedagogy that will make the critical difference to learners • We need categories within the ‘solution’ or ‘sequence of activities’ to help teacher: • Develop reciprocity and cooperation among students • Use active learning techniques • Give prompt feedback • Some accounts of learning activity sequences are quite elaborate

9. Representing learning patterns Structured text-based accounts , e.g. iCoper, Learning Designs Box-and-arrow diagrams for a sequence of activities, e.g. LAMS, iCoper, ISiS-ECTEL Text cannot be easily interpreted by a program. Computationally defined activities can be (e.g. LAMS). Therefore can be evaluated against theory.

10. Representing learning theories The Conversational Framework: developed to provide a representation of what it takes to learn in education based on the main design principles for learning and pedagogy drawn from the literature can test a learning design in terms of which aspects of the framework it covers.

11. The learner learning thinking, being supported, acting Listening/ Reading Guidance Articulating ideas Tuition Discussion TC LC OC LC Asking Questions Thoughts Others’ ideas Producing Reflection Adaptation Working to a goal Feedback Action plans Preparing Outputs TP Practice Imitation LP LP OP Investigating Others’ Outputs Revising

12. Learning theories represented in the Conversational Framework Listening/ Reading Guidance 1. Encourage contacts between students and faculty Articulating ideas TC LC OC LC Asking Questions Thoughts Others’ ideas Producing 2. Develop reciprocity and cooperation among students Reflection Adaptation Working to a goal Feedback Action plans Preparing Outputs 4. Give prompt feedback TP LP LP OP Investigating Others’ Outputs 3. Use active learning techniques Revising

13. Learning theories represented in the Conversational Framework Listening/ Reading Guidance Articulating ideas TC LC OC LC Asking Questions Thoughts Others’ ideas Producing Adaptation Reflection Reflection Adaptation Working to a goal Information Action plans Preparing Outputs TP LP LP OP Acting Others’ Outputs Revising Social constructivism

14. Learning theories represented in the Conversational Framework Listening/ Reading Guidance Articulating ideas TC LC OC LC Asking Questions Thoughts Others’ ideas Producing Adaptation Reflection Reflection Adaptation Working to a goal Information Action plans Preparing Outputs TP LP LP OP Acting Others’ Outputs Revising Instructivism

15. Learning theories represented in the Conversational Framework Listening/ Reading Guidance Articulating ideas TC LC OC LC Asking Questions Thoughts Others’ ideas Producing Adaptation Reflection Reflection Adaptation Working to a goal Feedback Action plans Preparing Outputs TP LP LP OP Acting Others’ Outputs Revising Situated learning

16. Representing technologies in support of learning Adaptive tutoring programs Wikis, blogs and Forums TC LC OC LC Thoughts Reflection Adaptation Virtual adaptive immersive environments User-generated content sites Action plans TP LP LP OP

17. Evaluating a learning design The Conversational Framework: can test a learning design in terms of which aspects of the framework it covers - IF we can link each LAMS activity (or similar computationally defined activity) in the sequence to part of the framework

18. LAMS – representing learning design as a temporal sequence of computationally defined activities To understand the processes within a system Chat Chat and Scribe Forum Forum and Scribe MCQ Notebook Noticeboard Q&A Resources&Forum Share resources Submit Voting

19. Mapping LAMS activities to theory via CF Noticeboard Listening/ Reading Chat and Scribe Guidance Articulating ideas MCQ Chat and Scribe TC LC OC LC Q&A Forum and Scribe Others’ ideas Asking Questions MCQ Forum and Scribe Producing Notebook Notebook Submit Voting Noticeboard Q&A Working to a goal Share resources Feedback Preparing Outputs Submit TP Voting LP LP OP Share resources Others’ Outputs Investigating Revising

20. LAMS – representing learning design as a temporal sequence of computationally defined activities To understand the processes within a system

21. Interpreting the quality of the design Evaluating the quality of the design System can interpret design as scoring: 4 out of 10 activities; No iterations; Time for each task is good; Sequence is good for ‘awareness’; less good for ‘understanding’ Noticeboard Listening/ Reading Guidance Articulating ideas MCQ TC LC OC LC Chat and Scribe Q&A Others’ ideas Asking Questions Forum and Scribe Producing Notebook Submit Voting Working to a goal Feedback Preparing Outputs TP LP LP OP Share resources Others’ Outputs Investigating Revising

22. Improving the quality of the design System can propose: Adding in designs or patterns that complete the other cycles; Checking that the sequence follows the motivating cycles of iteration Noticeboard Listening/ Reading Guidance Articulating ideas TC LC OC LC Chat and Scribe Q&A Others’ ideas Asking Questions Producing Voting Working to a goal Feedback Preparing Outputs TP LP LP OP Others’ Outputs Investigating Revising

23. Testing the framework The Conversational Framework: can differentiate between sequences in terms of coverage of the framework, amount of iteration, and effective ordering of activities But to what extent can it recognise the relative effectiveness of learning designs identified in the literature?

24. Intended learning outcome: To predict accurately the outcomes for a hypothetical experiment on memory [Schwartz and Bransford,1998] Learning activities Reading T sets a task goal Investigating Producing output Listening Working to achieve goal Actual outcomes Pattern 1 Contrasting cases text Analyse data S analyses data S produces graphs S listens to lecture S selects and predicts Very accurate MCQs High transfer to prediction Pattern 1 Contrasting cases text Analyse data S analyses data S produces graphs S listens to lecture S selects and predicts Very accurate MCQs High transfer to prediction Pattern 2 Theory and examples text Read text S reads text S produces summary S listens to lecture S selects and predicts Very accurate MCQs Much lower transfer to prediction Pattern 2 Theory and examples text Read text S reads text S producessummary S listens to lecture S selects and predicts Very accurate MCQs Much lower transfer to prediction Comparing learning design patterns Critical differences between patterns are not just the nature of the activity, but the internal relations between contents of activities in the sequence

25. Interpreting the quality of the designs 5. Listening 1. Reading Guidance System must be able to differentiate between producing graphs and summaries; And between providing Text and Data in the practice environment TC LC LC Asking Questions 4. Producing 4. Producing Adapting Reflecting 2. Working to a goal Feedback Text Data LP LP 3. Summarising 3. Investigating Revising

26. Representing the range of good pedagogy A System like LAMS can represent good pedagogy in terms of type and range of learning activity Cannot represent good pedagogy within the practice environment – simulation, model, digital library, data repositories, type of task and type of feedback Noticeboard Listening/ Reading Guidance Articulating ideas MCQ TC LC OC LC Chat and Scribe Q&A Others’ ideas Asking Questions Forum and Scribe Producing Notebook Submit Voting Working to a goal Feedback Preparing Outputs TP LP LP OP Share resources Others’ Outputs Investigating Revising

27. Summary Design principles in the literature Learning design patterns Representing learning theories as a framework for design Evaluating the pedagogy in a learning design Testing the evaluative capability of the framework Too general to be helpful for teacher-designers May miss categorising the critical pedagogy Makes it possible to relate theory to computational activities Needs mapping from framework to LAMS or similar system May not capture all the key differences in effectiveness

28. Evaluating learning designs through the formal representation of learning patterns Pedagogy operates both across and within activities • We need user-editable patterns for activity sequences • One role for learning technologists is to create the means for teachers to adopt and adapt pedagogic patterns for activity sequences – e.g. LAMS • One role for teachers is to these to adopt, adapt and share pedagogic patterns for activity sequences • We need customisable programmed patterns for activities • Another role for the learning technologist is to discover and create the means for teachers to adopt and adapt pedagogic patterns for practice environments – e.g. GLO tool, NetLogo Made publicly available by ALT under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales license http://creativecommons.org/licenses/bync/2.0uk/