Evaluation into the usability of a Virtual Learning Environment By Ian Cole (University of York) Dina Lewis (University of Hull) United Kingdom
Background • A partnership approach - NHS working with University of Hull and University of York. • NHS northern region e-Learning strategy. • Need to up skill workforce with knowledge and understanding of the potential of e-Learning to support work based practice. • We wanted to look at function and usability of a live course using a VLE. • FOLC course designed by University of Hull.
Facilitating online learning communities programme • 3 face-to-face workshops. • 7 online activities embedding e-theory in collaborative group work, case studies, presentations, use of web resources and e-learning pedagogy. • Course used the University of Hull’s Blackboard VLE. • 16 e-learning champions recruited. • 15 participants completed the course.
Participation rates were phenomenal 10 week programmeHits Weeks 1-3 12,593 Weeks 1-5 18,441 Weeks 1-10 30,464 Tutor 2,827 Most active participant 3,501 Least active participant 871 Usability test group 1,045 Average participant 1,773
Evaluation findings The programme was very successful in terms of: • The learning community, participation levels, motivation and commitment levels were extremely high. • Raising knowledge and understanding of key issues in supporting e-Learning. • Enabling participants to identify the potential of e-Learning to support work-based learning within their organisation.
Evaluation issues • Time was a key issue for all participants. • Need for coherent Senior Management strategy across organisations in terms of: protected work time for e-Learning, resources, venues, access to computers, IT support staff. • E-learning is not an add on it should be an integral mode of NHS core business.
Usability Study • Usability study conducted by the University of York. • Used Jakob Nielsen’s ‘framework of system acceptability’ where usability is a part of the usefulness of a system. • Learnability. • Efficiency of use, • Memorability, • Few and Noncatastrophic Errors • Subjective Satisfaction.
Usability Methodologies • Questionnaire Details: • 100 questions based on The Questionnaire for User Interaction Satisfaction (QUIS) tool. • Two part questionnaire. • a person specification. • a usability section. • Questionnaire based on Ben Shneiderman’s comprehensive criteria list for usability testing. • We modified Shneiderman’s scale slightly to include a 0 to 9 scale instead of 1 to 9.
Ben Shneiderman’s criteria: • Images. • Readability. • Use of colour. • Navigation. • Interactivity. • Learning to use the VLE including learning styles. • System capabilities. • General and overall reactions. • Accessibility.
Usability Methodologies • Cognitive Walkthrough Experiment. • Grounded in Lewis & Polson’s CE+ Theory. • 4 step information processing model: • The user sets a goal to be accomplished with the system. • The user searches the interface for available actions. • The user selects the action that seems likely to make progress toward a goal. • The user performs an action and evaluates the system's feedback for evidence that progress is being made. • Videoed the CWE tester interacting with Bb. • Results mapped against usability team & students responses.
5 person usability team. • The 15 FOLC students. • 1 Cognitive Walkthrough Experiment tester. Study Participates: Q85 – Using the VLE is: Frustrating Satisfying • Comment: • S: Had quite a few problems with access speed, which really frustrated me, as I didn’t have a lot of time to spend waiting for something to happen.
Study Results • Logging on to Bb was easy (8.1 mean). • The speed of logging in to the VLE is ok (5.0 mean). • Screen layout, organisation of info, navigation & text size were all between 5.0 & 7.1 mean. • Use of colour & images were 6.0 & 6.1 mean. • The instructions for using the Virtual Classroom are Inadequate (3.9 mean). • The Virtual Classroom interface is dull (4.8 mean). • Using the Discussion board is acceptable (above 5.7 mean) except search facility (4.5 mean).
Study Results • Getting lost when using the VLE is easy (4.3 mean). • Personalising the VLE screen is difficult (2.3 mean). • The flexibility of the VLE for dyslexic users (i.e. changing screen colours) is Inadequate (2.0 mean). • The flexibility of the VLE for visually impaired users (i.e. changing text size) is inadequate (3.5 mean). • The speed of the VLE is Too Slow (2.5.mean). • The VLE is unreliable (3.4 mean). All mean scores are totals.
Study Results • Overall Reactions to the VLE: • Terrible to Wonderful = 5.1 mean • Dull to Stimulating = 5.0 mean • Difficult to Easy = 6.3 mean • Frustrating to Satisfying = 4.1 mean • Rigid to Flexible = 4.8 mean • Inconsistent to Consistent = 5.2 mean
Conclusions & Recommendations: • Highly motivated students. • Problems with NHS networks being able to support VLE adequately. • Problems of speed accessing Bb server. • Bb should be more flexible regarding accessibility. • Better signposting & navigation in Bb. • More flexibility of the system for users and course developers.