Simulation as a tool for Computer Assisted Formative Assessment - First Aid as a case study

1. Simulation as a tool for Computer Assisted Formative Assessment -First Aid as a case study Andrew Young and Steven Cafferty Computer Science University of Salford

2. Formative Assessment • Very important • “That synonym of good learning” (Biggs 1999) • Very difficult to achieve • Large class sizes • Widening participation • Student-centered models such as PBL • CAA has good potential • but tends to focus on summative assessment

3. Our requirements for CAFA • Effective in assessing and encouraging learning • including deep learning (Bloom, SOLO) • Aligned to learning outcomes • Student-centered • Pedagogy driven, not technology driven

4. Our goal for CAFA • “To allow students to give themselves unlimited formative assessmentwithout the involvement of an academic” give themselves unlimited without the involvement of an academic”

5. What Formative Assessment isn’t • There are many informal (but often unhelpful) views of formative assessment • Giving a score that’s not used for grading • Giving the answer • Giving a model answer/worked solution • Going through the student’s solution

6. A pedagogical view • Using Kolb’s experiential learning cycle: Experience Experimentation Reflection Abstraction

7. So… Formative Assessment is • Generation of experiences... • …that encourage the student to go round the learning cycle… • …(or preferably that forces them to do so)… • …and that gives them feedback on the way they have done this

8. Deep learning • A series of “experiences” can be independent • which tends to lead to a “multistructural” learning outcome • or they can be dependent • which tends to lead to a “relational” (or better) learning outcome

9. Linked learning cycles • New “experiences” depend on previous “experimentation” • Reflection on a question implies reflection on the previous answers

10. Implementation • A hierarchical approach • the next question depends on previous answers • potentially exponential complexity • Simulation • generated “experiences” comprise a model of the world and a means of interacting with it • “reflection” and “abstraction” decide how to interact • the result of the “experimentation” is a modified model, to be used in future “experiences”.

11. Simulation • The “Economics” model • varying the parameters of how the model works is the key to interaction • The “Virtual Worlds” model • varying the state of the modelled properties is the key to interaction • Requirements • the world must be modellable • the model need not be deterministic • divergence depends on the degree of interaction

12. First Aid - an example application • Assessment of knowledge about cardiopulmonary resuscitation • aimed at refreshing the knowledge of people who have been trained but have not had the chance to practice • Bespoke simulator • BBC web site • illustrative example, not rigorous evaluation

13. First Aid simulator • STATE: condition of the patient and lifesaver • the initial condition and responsiveness can be randomised to ensure each simulation is different • INTERACTION: a list of things the lifesaver can do • still a little bit convergent • MODEL: • Lifesaver’s intervention affects patient’s condition • Lifesaver’s condition affects their effectiveness • AIM: Keep the patient alive until an ambulance arrives

14. BBC online site • A small example of a static assessment • …and a summative one • An informal experiment shows • people with no first aid skills score “better than chance” when they first take the test • their score rapidly approaches 100% if they repeat the test

15. Future work • Define a pedagogy to ensure it is “the right tool used in the right way” • Define the class of amenable problem • Equity - especially for special needs students and those with poor IT skills • Validity and reliability - are people just learning how to use the simulator? • Realism versus implementability and affordability

16. Can it be used for Summative assessment? • Problems • Validity and equity • Fairness - are weak students getting a “double whammy”? • Plagiarism • Potential • Assessment of “process” as well as “product”

17. Positive conclusions • A simulator allows students to give themselves formative feedback • It provides unlimited formative feedback • It doesn’t require academic moderation • It should be possible to assess deep learning

18. Negative conclusions • High up-front development costs • A lot of the knowledge and interaction is domain specific

19. Thank you • Any questions? • Any comments? • Any examples of domains where this will be useful? • Any examples of domains where this will NOT be useful?