1 / 32

CAA in UK HE:

CAA in UK HE:. The State of the Art Bill Warburton University of Southampton Email: wiw@soton.ac.uk. Research & Graduate School of Education. The 2003 survey. Introduction Demography Findings: Question banks CAA software Pros & Cons Question types VLEs & MLEs Interoperability

teva
Download Presentation

CAA in UK HE:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CAA in UK HE: The State of the Art Bill Warburton University of Southampton Email: wiw@soton.ac.uk Research & Graduate School of Education

  2. The 2003 survey • Introduction • Demography • Findings: • Question banks • CAA software • Pros & Cons • Question types • VLEs & MLEs • Interoperability • Summary Research & Graduate School of Education

  3. The 2003 survey • Why do a CAA survey now? • Increasing prevalence of web-based delivery • Pervasiveness of Virtual Learning Environments (VLEs) • Emergence of Managed Learning Environments (MLEs) • Emergence of Interoperability standards e.g. IMS/QTI • Four year gap e.g. between 1999 & 1995 CAA surveys Research & Graduate School of Education

  4. The 2003 survey • Similarities with the (1999) CAA Centre Survey* • For congruence, a superset of the 1999 survey • Dissimilarities with 1999 survey • Extra questions on VLEs, MLEs & Interoperability • Targeted CAA enthusiasts rather than non-users • Delivered purely by web-based CAA system Research & Graduate School of Education *Bull, 1999; Bull & McKenna, 2000; Bull & Hesketh, 2001

  5. The 2003 survey: Who were the respondents? • Targeted people who were likely to be using CAA • Incentive- prize draw of two £25 book tokens • Survey questionnaire was distributed using three JISC mailbase lists: • CAA • TOIA • ALT • 50 responses received by initial deadline of 18th May • Will be issued more widely Research & Graduate School of Education

  6. The 2003 survey: Who were the respondents? • CAA Involvement: • A constituency of enthusiasts Research & Graduate School of Education

  7. The 2003 survey:Job Function • Little more than half the 2003 respondents were academics (1999: 77% were academics) • An increased proportion of pedagogic & ICT support Research & Graduate School of Education

  8. The 2003 Survey: Who supports CAA now? • CAA support is largely centralised • Even where support has been devolved to faculties / departments, there is usually a central CAA service Research & Graduate School of Education

  9. The 2003 survey: Who uses CAA now? Usage by Subject : broadly the same as 1999 Research & Graduate School of Education

  10. 2003 Survey: The use of question banks • Nearly 60% said they now use question banks • Nearly 90% would be interested in sharing questions Research & Graduate School of Education

  11. The use of question banks • Pros of question sharing- shared loads: • Authoring • QA • Editing • Cons of question sharing: • Initial time penalty • difficulty of applying rigorous QA standards across multiple authors & institutions • requirement for multiple formats to suit all users Research & Graduate School of Education

  12. The use of question banks • Example in Science: • The e3an project developed an item bank of more than 1500 questions for use in Electronics and Computer Science • Example in Humanities: • A question bank has been developed to pre-course assess modern language students Research & Graduate School of Education

  13. 2003 Survey: What CAA software is used in HE now? • Commercial CAA systems are more dominant • Many respondents used multiple CAA systems

  14. 2003 Survey: What CAA software is used in HE now? • Split between users of Commercial and Open systems • Commercial systems- traditionally seen as expensive but easier to use • Non-commercial systems e.g. TRIADS- free, but support-intensive Research & Graduate School of Education

  15. 2003 Survey: Advantages of different CAA software Research & Graduate School of Education

  16. 2003 Survey:Disadvantages of different CAA software Research & Graduate School of Education

  17. 2003 Survey: Main benefits of CAA • Little different from 1999, considering small sample • 1999 survey: • Speed of feedback to students • Statistical analysis of results • Automatic marking • 2003 survey • Speed of feedback: “Available 24/7 - with instant feedback” • Speed of automatic marking: “50 pieces of coursework marked in a flash” • Formative use promotes student learning: “students often retake quizzes and improve their score”

  18. 2003 Survey: Main disadvantages of CAA • There appears to have been a shift- Learning curve and up-front investment are now seen to be the most prominent disadvantages • 1999 survey: • access to and reliability of hardware and software • amount of time needed create and organize delivery • difficulty of writing good questions • 2003 survey: • amount of time needed create and organize delivery • access to and reliability of hardware and software • difficulty of writing good questions Research & Graduate School of Education

  19. 2003 Survey: Does CAA save time? • The 2003 respondents were evenly split regarding the time saved by CAA Research & Graduate School of Education

  20. 2003 Survey: Does CAA save time? • Surprisingly, a large majority of those who reported productivity benefits did not see a corresponding shift in workload to other staff Research & Graduate School of Education

  21. 2003 Survey:Formative vs. Summative testing • There appears to be a definite shift towards formative use: less than half the 2003 tests were summative (1999: most tests were summative) • Diagnostic use is emergent: other formative usage is long-established Research & Graduate School of Education

  22. 2003 Survey:Question types • 1999: MCQ & MRQ most popular followed by text and numeric input, then graphic hot spot and other types • Little appears to have changed in 2003: MCQ still top Research & Graduate School of Education

  23. 2003 Survey: Who is doing CAA? • Moore's Gap has yet to be crossed: CAA appears still to be largely the province of ‘early adopters’ • Will CAA follow VLEs and become mainstream technology? Research & Graduate School of Education

  24. 2003 Survey:The growth, influence and relevance of VLEs • Have VLEs crossed Moore’s Gap? Are now pervasive • Most respondents (81%) had implemented site-wide VLEs • It seems that VLE quiz tools give users a taste for CAA • The uptake of VLEs may foreshadow the uptake of CAA Research & Graduate School of Education

  25. 2003 Survey:The growth, influence and relevance of VLEs • VLEs are seen as a largely positive (60%) influence on the uptake of CAA • Cautionary note: Two reports that VLEs apparently discouraged the use of CAA Research & Graduate School of Education

  26. 2003 Survey:Interoperability & VLEs • The majority of VLE quiz users wanted to export their quizzes to a dedicated CAA system • One respondent wanted to import CAA items and tests into a VLE Research & Graduate School of Education

  27. 2003 Survey:tension between VLE quizzes and CAA • Many respondents attempting to do CAA using VLE’s Quiz engine • Most commented unfavourably on VLE Quiz engines: “…testing facility is very limited, but is all we have available” • Cultural & technical differences between VLEs & CAA: • Words do not always have common meanings- • Blackboard refers to a quiz as an ‘item’ • In CAA an ‘item’ is usually a single question • VLE quiz items/tests often restricted to proprietary formats, & may be less secure Research & Graduate School of Education

  28. 2003 Survey:The emergence of MLEs • More than two thirds of respondents’ institutions have stated an intention to develop an MLE • However, the concept of an MLE is contended (JISC 7/99 projects) Research & Graduate School of Education

  29. 2003 Survey:Interoperability • The majority of HE users thought interoperability to be important Research & Graduate School of Education

  30. 2003 Survey:Interoperability • Although most HE users thought interoperability to be important, awareness of interoperability standards varied widely Research & Graduate School of Education

  31. Summary • Superset of 1999 survey + extra questions on VLEs, MLEs & Interoperability • Demography: More support staff involved than in 1999 • Question banks: Desirable but rare • CAA software: Commercial CAA systems dominant but inflexible • Pros: Little changed from 1999- time savings • Cons: Learning curve now more significant than cost • Question types: MCQs & MRQs still predominant • VLEs: have crossed Moore’s Gap and appear to drive CAA • MLEs: Most HEIs intend to implements MLEs (but concept of MLE contended) • Interoperability: thought to be important, but awareness of standards varies Research & Graduate School of Education

  32. End of presentation • Questions? • Question banks: desirable but how much effort? • CAA software: What features do people most want? • Pros & Cons: at what point do the pros outweigh the cons? • What question types do people most use? Most want? • What roles do Interoperability, VLEs & MLEs play in promoting CAA? • Do Interoperability, VLEs & MLEs drive CAA or vice-versa? • When will CAA cross Moore’s Gap? Research & Graduate School of Education

More Related