Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand
This presentation is the property of its rightful owner.
Sponsored Links
1 / 24

Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert PowerPoint PPT Presentation


  • 53 Views
  • Uploaded on
  • Presentation posted in: General

Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert LTEU, Canterbury Christ Church University. Outline. The history of online summative tests at CCCU using short-answer questions delivered online A reflection, 2005 – 2009 What do we want from CAA?

Download Presentation

Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand

Glenis Lambert

LTEU, Canterbury Christ Church University

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Outline

The history of online summative tests at CCCU using short-answer questions delivered online

A reflection, 2005 – 2009

What do we want from CAA?

How do we match up to this today?

The way forward

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Background: Canterbury Christ Church University

Faculties of Health and Social Care, Education, Undergraduate Programmes +++

Large numbers of students on placement

Diverse student body

Growing numbers of collaborative programmes

Multi-location campus

Growing student numbers

Central Learning and Teaching Enhancement Unit with Learning Technologists for each Faculty

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

2005

What we did

Bought QuestionMark/Perception

Mapped with existing University policies and managed conflicts

Using BS ISO/IEC23988:2007 Wrote a policy document

Set up staff procedures and guidance with full documentation for staff

What happened in the pilot phase

Over the academic year 8 end of course exams in 18 sessions were successfully delivered, totalling 660 individual examinations. 5 of these were organised and delivered by the departments, three by the examinations office.

2005

2006

2007

2008

2009

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

2006

What we did

  • Slight increase in number of tests

  • Consolidated procedures

    What went wrong

  • Lack of staff confidence in online environment led to high support demand

  • Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student

  • Human error!

  • What we Learned

  • Stakeholder confidence in support and procedures help develop practice

  • Buy-in at all levels is essential

  • Clear fall-back strategies are essential

  • BUT:

  • Exit surveys revealed student satisfaction with online examinations

  • All of the fall-back procedures worked well

  • The systems/software performed well

2005

2006

2007

2008

2009

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

2007

What we did

  • Delivered over 5,000 summative tests and a

  • few other surveys.

  • Support load on LTEU began to be unsustainable.

  • What went wrong

  • Staff still not engaging with processes and procedures

  • Infrastructure made process expensive

  • Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student

  • Human error!

  • “Last minute” culture led to increased support demand

  • Infrastructure began to creak!

2005

2006

2007

2008

2009

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

2008

What we did to try to manage the support load

  • Put an embargo on all new tests and set out to reinforce our policies by working with stakeholders to devolve some of the activities carried out by LTEU

  • Bought the Blackboard Bridge to reduce the administrative load

  • Encouraged more academics to do “open” assessments which reduced the infrastructure and invigilator load, but not the support load, although this changed its emphasis. (Bertolo and Lambert, 2007)

  • What we learned

  • Our ambition to control the testing environment was probably never going to happen

  • It was going to take a long time to get all stakeholders on board

  • There needed to be a central agency to co-ordinate all aspects of the process

  • Support time for CAA was always going to be high

  • Infrastructure/development costs were an ongoing commitment

  • We had yet to muster a persuasive argument to get total institutional buy-in

2005

2006

2007

2008

2009

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

2008

What went well

  • We have rarely had to resort to “fall-back” positions, e.g. paper

  • Our quality procedures have been robust and applied throughout

  • Our planning procedures have raised stakeholder confidence

  • Documentation produced for staff has been welcomed resulted in clear definitions of responsibilities

  • Staff development has led to improvements in all areas

2005

2006

2007

2008

2009

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

2009

  • What we learned

  • We needed to think carefully about what we were doing and whether we should continue to do it.

2005

2006

2007

2008

2009

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

What do we require from CAA?

Report on eAssessment Quality (REAQ)

Overview

There is wide consensus that e-assessment promises many potential benefits to the JISC community, ranging from lowered costs, higher productivity, and faster feedback, through to assessments which are more accurate, more detailed, and more robust under critical scrutiny and audit. There is equally wide concern that these benefits seem relatively slow to materialise, and may be negated by poorer quality assessment which fails to reliably provide acceptably valid measures of actual student achievement and capability, particularly at the higher levels of Bloom’s taxonomy of the cognitive domain. The issues confronting staff concerned with the quality management of e-assessment suggest a pressing need for evidence, for information, and for guidance.

http://www.jisc.ac.uk/whatwedo/projects/reaq.aspx

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

What do we require from CAA – the Institution’s view?

Assessed tasks should attract sufficient student time and effort

Tasks should engage the student in productive learning activity

Students' effort should be spread as evenly as possible

Orientation to the task should help students to perceive the demands of an assessment task

Communication through guidance provided should offer clarity on expectations

Sufficient and timely feedback should be provided

Feedback should focus on learning not on the marks or the student

Feedback should be closely linked to assessment criteria

Feedback should be given at a level accessible to the student's sophistication

Students should be required to receive and respond to feedback

http://www.canterbury.ac.uk/support/learning-teaching-enhancement-unit/assessment/assessment-handbook/pages/section1/11-3-strategies.asp

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Can CAA ever be “fit for our purposes?”

Develops students’ abilities to evaluate own progress, direct own learning

Emphasises authentic and complex assessment tasks

Uses high-stakes summative assessment rigorously but sparingly

Offers “low-stakes” confidence building opportunities and practice

Is rich in formal comment, (e.g. tutor comment, self-reflective logs)

Is rich in informal feedback (e.g. peer review, collaborative project work)

Above all, it should enhance learning.

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Quality Assurance Procedures

‘Openness to challenge is a critical cultural necessity for good risk management and compliance – it is in fact more important than any framework or set of processes’.

Paul Moore, Evidence to Treasury Select Committee (February 2009)

“Over the last twenty years, the higher education sector has been subject to structural and cultural changes associated with a focus on accountability and value for money which have profoundly affected academic life…..  In this new reality the term ‘quality’ became synonymous with data collection, performance scrutiny and a massive increase in bureaucracy. ”

George Gordon and Catherine Owen, Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners  

Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland (2009)

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

What are the risks?

Risks involved in the continuing growth of CAA:

High costs of quality-assurance not met by institution

High costs of reliable delivery not met by institution

Concentration on “box ticking” accountability does not lead to enhanced learning

Risks of NOT using or encouraging the growth of CAA:

Opportunity for enhancing learning and feedback lost

Students’ expectations of HE not met (Dermo 2009)

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

What we want from CAA

Embedding

Embedding

  • In addition:

  • That it is used to provide speedy feedback to students

  • That it is used as a means of enhancing learning

Strategy

Strategy

Policy

Policy

Robust

Robust

Reliable

Reliable

Fit for Purpose

Fit for

purpose

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Where we are at present

Where people have used CAA it has become embedded in their practice, but summative use isn’t leading to formative use.

Embedding

There is a growth in the strategic use of CAA, but largely to reduce marking load.

Strategy

Policy documents are now in existence, but they need reviewing in the light of support demand.

Policy

We are confident that our policies and procedures do result in adequately assured assessments where they are applied, but the infrastructure and support services are sometimes more fragile

Robust

The software and systems have proved to be reliable. Infrastructure and the application of policy and procedures less so.

Reliable

“Quick Burn” implementation (Warburton 2009) leading to some questionable practice

Feedback not built in to tests, little formative testing

Fit for

purpose

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Conclusions

The support and maintenance costs of quality-assuring CAA are high and ongoing

It is easy to quality-assure CAA if it is only seen as a method of providing accountability for assessment practices and procedures

If the major requirement of CAA is to enhance learning, it can be cost-effective, when “opportunity costs” are factored in (Ridgway et al. 2004).

To achieve the benefits of CAA for learning enhancement staff development has to be the central strategy for assuring quality.

A “student satisfaction” ethos needs to be central to mitigating failures

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Forward plans

  • Raising awareness through Enhancement Theme on Assessment and Feedback

  • DEBUT staff development project

  • Work with staff on planning of new courses via FQOs

  • Development of e-learning strategy

  • Benchmarking e-learning, including use of CAA

  • Inclusion of policy and procedure documents in staff development pack

Embedding

Strategy

Policy

Continue to work with all stakeholders to emphasise the importance of team structures

Staff development for all stakeholders including Examinations Office, User Technology etc.

Robust

In the absence of more resources, designate a number of rooms as testing labs to make infrastructure support more efficient

Make sure all stakeholders know about fall-back procedures

Reliable

Maintain central consultancy role in LTEU for construction of questions and tests

DEBUT staff development project

Fit for

purpose

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

References

Assessment Handbook, Canterbury Christ Church University

http://www.canterbury.ac.uk/support/learning-teaching-enhancement-unit/assessment/assessment-handbook/pages/section1/11-3-strategies.asp

Bertolo, E & Lambert, G: 2007 Implementing CAA in Chemistry: A Case Study

Procedings of 11th CAA Conference, Loughborough University 2007

Dermo, J. (2008) “Implementing Online Assessment: Finding the Right Path for an HE Institution” in Ladwa, A (ed) E-Learning in HE available online at http://www.rsc-yh.ac.uk/Documents/HEbooklet2_000.pdf (last accessed June 2009)

Gordon, G & Owen, C, 2009,Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners

Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland

Parsons, R, (2004) Ensuring Quality and Efficiency with Online Assessments JISC

http://www.enhancementthemes.ac.uk/documents/events/20040416/Parsonspaperrevised.doc.

Report on eAssessment Quality (REAQ), JISC 2009

http://www.jisc.ac.uk/whatwedo/projects/reaq.aspx Accessed June 2009-07-02

Warburton 2009, Quick win or slow burn: modelling UK HE CAA uptake

Assessment & Evaluation in Higher Education, 1469-297X, Volume 34, Issue 3, Pages 257 – 272

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

What we want from questions and tests

That online assessments are viewed as routine, in the same way as presentations, paper tests etc.

Embedding

That short, formative online tests are a routine part of assessment strategies

Strategy

That policies and procedures are routine elements of assessment procedure

Policy

That all questions and tests are valid and peer-reviewed

Robust

That staff can rely on support and staff development in the preparation of assessments

Reliable

That all elements are “fit for our purposes”

Fit for

purpose

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

The question question: a digression.

Stage 1:

This is a duck: True or False?

Stage 2:

This bird is:

A cootc) A duck

A swane) A diver

Stage 3:

Identify this common duck and match it with its species

Mallardi) Marmaronetta

Pochardii) Netta

Tealiii) Aythya

Glenis Lambert LTEU June 2009


Aligning the ducks sustaining the quality of caa in a period of growing demand glenis lambert

Barriers to improving question quality

Many academic staff operate in a text-based environment and find the black and white nature of computer-marked questions challenging

Learning technologists do not have detailed subject knowledge

Staff development to enable the development of question-setting skills takes time.

Glenis Lambert LTEU June 2009


  • Login