Gavin Sim, Naveed Malik, Phil Holifield

1. Strategies for large-scale assessment: an institutional analysis of research and practice in a virtual university Gavin Sim, Naveed Malik, Phil Holifield

2. Introduction • Background of VU • Infrastructure • Strategy • Admissions • Assignments • Examinations • Future Strategy • Conclusions

3. Background • Virtual University Government Funded Project • Courses delivered via satellite television and a Learning Management System (Intralearn) • Utilise lead academics to develop courses • Students attend Learning Centres across Pakistan to view programmes and use the Internet • Evidence that the needs of learners in remote areas can be served via local learning centres (Mayes, 2001)

4. Pak Institute of Management & Computer Science – Lecture

5. Pak Institute of Management & Computer Science - Lab

6. Background - Cont. • Current Cohort 1700 capabilities of 30750 • Universities in Pakistan full year course followed by exams • VU this strategy not feasible, students need to be able monitor progress and obtain feedback • Marking time would be 212 days for typical exam scripts • New Strategy required utilising the technology

7. Infrastructure • Guidelines highlight need to identify technical issues (Stephens, Bull et al. 1998) • VU identified, distributed cohort - connectivity, scalability, security & authenticating users • Academic Intranet • Courses and assessment delivered via the LMS • Intralearn requires a ColdFusion backend and a DBMS which Microsoft SQL Server is used • Intralearn and SQL-Server are hosted on separate servers creating a 3-Tier approach

8. Infrastructure - Cont.

9. Infrastructure - Cont. • Scalability an Issue • Max bandwidth between Learning Centres and VU Hub 2Mbps can increase to 155Mbps • The LMS needs an adaptable architecture • Custom built software has been developed for submitting the assignments (http upload). • Asynchronous assessment is required due to the variability in connectivity between centres • Students can use any software - this is then encrypted, renamed and forwarded to the tutor for grading

10. Infrastructure – Cont.

11. Infrastructure - Cont. • Delivery of exams - security issues • Security flaws in sending emails such as authentication and encryption (Hatton, Boyle et al. 2002) • Due to student numbers an asynchronous approach required • Software developed and deployed at centres • Exam sent to centres encrypted and placed on local server • Same file returned encrypted after the exam

12. Strategy - Admissions • Admissions scheduled for spring and autumn • Test administered to 5000 students comprising MCQ and essay questions. • Marking outsourced to National Testing Centre • Found no advantage in using this approach due to bias for those exposed to testing before • Test anxiety could adversely effect their grades (Cassady & Johnson 2002) • Admissions based on public examination results

13. Strategy – Assignments • Account for 15% of marks and students usually have between 5-10 per semester. • Regular testing is vital (Croft, Danson et al. 2001) • A series of bad marks can lead to a downward spiral in learning and confidence (Taras, 2002) • The assignments are structured to build student confidence • CS101 first assignment send an email and use the discussion board

14. Strategy – Assignments Cont. • Submission of assignments deadline plus one day • Academics are responsible for designing the assignments • Essay style formats usually used rather than objective testing • Tutors responsible for marking 250-300 assignments per week • Number of tutors is proportional to the number of students enabling scalability

15. Strategy – Assignments Cont. • Solutions are posted on LMS no feedback is provided • Self-reflection against a model answer enables students to gain personal insight of their understanding (Peat, Franklin et al. 2001) • Can ask questions concerning assignments on moderated discussion boards

16. Strategy - Exam • Two exams per module mid-term 35% and final exam 50% • Examiners are responsible for authenticating student ID at learning centres • Exams consist of both MCQ and Essay style questions • MCQ issues surrounding guessing (Harper, 2003) • Essays subjectivity with different markers

17. Strategy – Exam Cont. • Having varied question styles help produce fairer test • MCQ marked automatically whilst essays are marked by tutors • The students type their essays improving legibility for the tutors • This method enables the full range of cognitive skills to be tested as defined by Bloom (1956) • Debate whether objective testing can test higher cognitive skills in all subject domains

18. Strategy - Exams Cont. • In the majority of instances Synthesis and Evaluation promote divergent thinking and answers cannot be determined in advance (Bloom 1971) • Higher Cognitive Skills tested in Linguistics (Reid 2002) • Results are posted on the LMS • Any queries are directed to staff within the VU

19. Future Strategy • Group projects planned for final year • Students will work with others outside their geographical area • Tutors responsible for grading projects • Explicit guidelines will need to be drawn up • Ways to overcome gaps in face-to-face presentations and lab demonstrations are sought

20. Conclusions • Infrastructure and strategy need to be developed in parallel • A strategy of combining CAA and tutor marked assessments is feasible • The scalability of the strategy and infrastructure should enable successful future expansion • Future analysis and research in the field of CAA may produce solutions that overcome some of the drawbacks

21. Questions