The formative use of e-assessment Some early implementations, and

1. The formative use of e-assessment Some early implementations, and Suggestions for how we might move on Andrew Boyle

2. Initial stuff – rationale, definitions, etc.

3. Introduction • Early predictions re e-assessment • e-assessment will lower barriers between formative and summative assessment • Early years of e-assessment coincided with renewed interest in formative assessment (FA) • Now e-assessment needs to move to ‘mature’ phase • Literature review • Important role in FA research • Attempt to summarise where we have got to with formative e-assessment (eFA) • Group implementations to establish coherence • Move towards critique

4. Definitions • FA • All those activities undertaken by teachers, and/or by students, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged • Contrasted with summative assessment • Summarises learning • Used for recording and reporting the amount of learning but not for feeding back into learning • Important strand in both school and HE assessment research • Slightly different emphases • e-assessment includes • Tests that are delivered on-screen • e-portfolios • Electronic discussion boards, forums and so on

5. Research aims • Identify types of implementations used frequently by researchers and developers working in the field • Suggest ways in which the body of research evidence might be expanded • Underlying aim to delineate those areas where eFA provides a distinctive input • Balance ‘e-assessment as transformative’ and more sceptical views

6. Method and scope • Review is inclusive, rather than excluding • ‘Best evidence synthesis’ • Seeks authentic, faithful and convincing results • Does not comply with one or more ‘objective’ criteria • A thorough review of eFA literature • Backed up by a selective review of FA literature • Not a general review of e-assessment

7. Background findings re ‘plain’ FA

8. FA research is about several things • Interaction • Teacher-learner; learner-learner • Feedback is central • Speech – questions and answers • Comment-only marking on written work • Focus on features of the work, not the individual • Permit learner to ‘close the gap’ between current and desired performance • Features of FA items and tests relatively little studied • Can have more (or less) than one correct answer • Distractors explicitly connected to incorrect or incomplete conceptions • Item responses provide clues to effective action

9. Claims about FA • Associated with learning gains • Claimed to be one of the strongest associations between educational intervention and learning gains • Some question about whether all reported gains are caused by the FA intervention • Disseminating messages to practitioners key • Evidence that changes in practice have been patchy

10. Organisation of eFA findings

11. Areas for further work Key issues Findings

12. Finding 1 Electronic technologies provide a range of new tools that classroom teachers can use to create formative assessments to suit their and their students’ needs. • Variations on the theme of MCQ • Sophisticated tasks – rich in interactivity and multimedia • Test designs specific to e-assessment • e-portfolios to closely integrate formative and summative assessment • Communications tools such as electronic discussion boards and forums for self- and peer feedback in e-learning courses

13. Characteristics of formative & summative items • eFA implementations take item and task types from summative assessment • Cf. Wiliam’s characteristics of good formative items • Need to build on research into FA item types • Establish distinctive characteristics of ‘good’ eFA items/tasks

14. eFA as exam revision • FA research emphasises several facets • Many eFA implementations equate FA with revision or practice testing • Revision not a bad thing per se • Can reduce test anxiety • Support distance learners • But is a reduced concept compared to totality of FA

15. Need to write own test questions • Early implementations of eFA often involve innovators developing own questions • Realistic for all teachers to write their own questions? • If teachers select from pre-written banks of questions, is anything lost? • Would they be able to tailor the formative assessment to their learners’ needs? • What sort of skill is selecting balanced formative tests from an item bank?

16. Finding 2 e-assessment functionality permits formative feedback to be given in variety of ways not possible in plain FA. • Variety of feedback methods in eFA test applications • Advantages of e-portfolios for providing feedback • Encourages teachers and learners to interact about drafts • Process of generating work forefronted, rather than merely concentrating on the final product • Communications tools can give feedback varied by: channel, recipient, formality, etc. • Can target learners of different styles • E-learning technologies can facilitate giving of feedback to distance learners • Contribute to online discussions (peer feedback) • Reflective journal (self-assessment) • Evaluate peers’ work (either formally or informally)

17. Principles for feedback in e-tests • Need to systematise understanding of e-test feedback • Does ‘comments not grades’ apply to eFA? • If so, one of the most obvious benefits of e-assessment (rapid right/wrong info) is removed. • Is engagement with rich media and interactivity synonymous with deep learning? • Could learners be distracted – clicking through sites without really truly processing content? • Could we relate feedback to students’ learning styles?

18. Practicality of e-portfolio feedback • Need logistical studies that investigate practicality of using e-portfolios • Ensure that teachers can provide high-quality feedback and potential is not lost • ICT elements of portfolios should reduce burden

19. Impact of cultural factors in e-communication • Students giving and receiving feedback need to understand cultural norms • Students working at distance may bring different prior assumptions to feedback • Culture can also mean academic culture • Concern if students have not internalised academic norms • Requirement to style switch between varying e-discourses could make it harder for some to internalise ways of writing

20. Finding 3 eFA applications can be used remotely. This provides a resource which is not easily replicated via pencil-and-paper materials. • Advantages of asynchronicity for undergraduates: • Reduced examination stress • Popular and motivating • Free up teacher time • Allow students to increase self-regulation for tertiary study • Asynchronous online discussions facilitates enhanced reflection • Areas needing further clarification • Effect of learning styles and motivation of usage • Already self-regulating students benefit more • Usage differs between intrinsically interested & ‘pragmatists’ • Clarify most common usage patterns

21. Better studies of learning gains • Many papers assert attainment benefits of eFA • Esp. for self-access use • Claims undermined by research design • Small cohorts • Test difficulty not equated • Variables confounded • Does eFA provide attainment benefits over and above ‘plain’ FA? • It might – there are also claims that use of ICT in schooling is associated with improved attainment

22. Self-access for primary and secondary students? • eFA as self-access strongly associated with tertiary • What issues for self-access use by school pupils? • Greater obligation on teachers to moderate feedback • Role of parents in online learning • Effective use to support ‘personalised learning’?

23. Superordinate ‘key issues’

24. Define how the ‘e’ goes beyond plain FA, AND • Define genuine distinctiveness of eFA as opposed to summative e-assessment • Use of instruments by practitioners • Provision of feedback • Need to demonstrate learning gains • eFA must not be a reduced version of ‘plain’ FA • eFA as exam revision • Provision of feedback • Need to understand how teachers and students adapt to novel roles that eFA requires • Cultural demands of using e-communication tools • Strong element of independent working and self-assessment • Beware of how eFA may impose novel burdens on teachers and students • Requirement of teachers to write own test questions • Need to provide manageable systems for feeding back via e-portfolios

25. Thanks for listening. • Questions/comments …

26. Features of FA items and tests • Can have more (or less) than one correct answer • Distractors explicitly connected to incorrect or incomplete conceptions • Item responses provide clues to effective action