Assessment design, pedagogy and practice: What do ‘new’ lecturers think?

1. Assessment design, pedagogy and practice: What do ‘new’ lecturers think? Lin Norton, Bill Norton, Lee Shannon and Frances Phillips ISSOTL 2009

2. Acknowledgements • Research has been funded by the Write Now CETL • The researchers include: • Ola Aiyegbayo (formerly of Liverpool Hope University) • James Elander (Derby University) • Katherine Harrington (London Metropolitan University) • Frances Phillips (Liverpool Hope University) • Peter Reddy (Aston University) • Lee Shannon (Liverpool Hope University) ISSOTL 2009

3. Background: Is there a problem with assessment? • Authentic assessment can be defined as assessment that is pedagogically appropriate- it frames students’ views of HE, it has a major influence on their learning and it directs their attention to what is important (Boud & Falchikov, 2007) But… • In the USA, many institutions tend to adopt assessment approaches more traditional than those recommended in the literature (Peterson & Einarson, 2001) • In the UK context, Rust (2007) asserts that many assessment practices are poor and not underpinned by pedagogical literature. • There seems to be a gap between SOTL and what it has to say about assessment and actual assessment practice ISSOTL 2009

4. So what is currently happening in HE? • Widely held view that assessment should be for rather than of learning (Black & Wiliam, 1998; CETL:AfL http://www.northumbria.ac.uk/cetl_afl/) • But… is this what lecturers(faculty) think- particularly new lecturers, and, if so, are they able to put their beliefs into practice? • And… what part do taught programmes for lecturers have to play? ISSOTL 2009

5. What part does a university teaching programme play? • An initial interview study was carried out with 10 new lecturers taking a university teaching course (Norton, Aiyegbayo, Harrington Elander & Reddy, in press) • Our findings suggested that the course was generally perceived as useful and influenced the way that the new lecturers thought about assessment: • ‘The assessment module [in the course] revealed to me that assessment is not just about assessing how much people know but should be used as a tool in learning. This was an eye opener for me.’ (G) ISSOTL 2009

6. Interview study cont.. • When asked if they could change assessment practice easily, 6 said no and 4 said yes but with reservations: • Not really because it is set in stone in the module proposal. You have to jump through many hoops if you are going to change the assessment techniques.’ (A) • ‘No! I get the impression that they are set in stone…. I think that hurdles of going to various panels to have your module changed puts people off…I get the impression from talking to colleagues that the process is long-winded and bureaucratic.’ (C) • We concluded that what new lecturers learn about assessment does not always readily translate into practice. This may be caused by a complex interaction of institutional, departmental and individual factors (Becher & Trowler, 2001; Fanghanel, 2007), which we have called ‘constraints’ • We wanted to explore these constraints further, using a questionnaire approach ISSOTL 2009

7. Current research study • First of several large scale quantitative studies carried out by the Write Now CETL research team investigating lecturers’ views on assessment, marking & feedback. • This study focuses on lecturers who were taking, or had recently taken, a university teaching course and their views about assessment design. Aims • To analyse whether there were any potential constraining factors hindering lecturers from putting what they had learned about assessment into practice. • To develop the Assessment Design Inventory (ADI) as an instrument ISSOTL 2009

8. Development of the ADI • The Assessment Design Inventory (ADI) comprises 40 items developed from over 600 items constructed from 84 interviews with lecturers (new and experienced) from 18 disciplines, in 5 institutions. • An earlier version of the ADI consisting of 68 items was piloted on a sample of 29 lecturers from one UK university • The ADI was sent out as an online survey aimed at those lecturers currently on HE teaching courses or who had recently (in the last 2 years) completed such a course. • Access to respondents was by invitation via HE teaching course leaders at all UK institutions (150). • Around half the institutions agreed to circulate the invitation to their course participants ISSOTL 2009

9. Participants • 586 fully completed inventories ( 325 females, 256 males, & 5 not specified ) • 80 discipline areas and 66 institutions from England, Ireland, Scotland & Wales • 276 were currently on a HE Teaching course; 308 had already attained an HE Teaching qualification. • University teaching experience ranged from less than a year to 26 years: ISSOTL 2009

10. Assessment Design Inventory(ADI) • Initial exploratory analysis suggests 3 underlying factors: • Constraints e.g. ‘Changes to my assessment design are hindered by external factors (e.g. cost, high student numbers, time).’ • Desirable practice e.g. ‘Involving students in the assessment design would encourage them to engage in the assessment task’. • Purpose e.g. ‘The main aim of assessment is to document student achievement to employers.’ ISSOTL 2009

11. Constraints • Results will be presented for the 12 items on the ADI that specifically asked about: • the potential constraints on turning assessment design philosophy into assessment practice • We were particularly interested in looking for any effects on responses relating to: • participation on HE teaching courses • length of experience in teaching in Higher Education ISSOTL 2009

12. The effect of the University teaching programme on new lecturers’ beliefs about assessment • My experience on the University teaching programme has changed my views on assessment practice. 74% agree16% disagree10% uncertain [Mean=3.80, SD=0.99] No effect for teaching qualification or length of experience ISSOTL 2009

13. How do new lecturers feel about innovating assessment? (1) • New assessment methods are needed to improve current practice. 69% agree17% disagree14% uncertain [Mean=3.67,SD=0.95] Lecturers who had already attained a teaching qualification agreed more strongly than those currently taking a teaching course • There is little incentive for lecturers to innovate in their assessment practice. 61% agree32% disagree7% uncertain [Mean=3.40,SD=1.18] Lecturers with 11+ years experience were less likely than those with less than 2 years experience to agree that there was little incentive to innovate ISSOTL 2009

14. How do new lecturers feel about innovating assessment? (2) • Innovation in assessment is not well received by students. 31% agree52% disagree16% uncertain [Mean=2.76,SD=1.15] Lecturers with 11+ years experience were more likely than those with less than 2 years experience to disagree. • I feel my ideas are valued when developing assessments. 65% agree16% disagree20% uncertain [Mean=3.53,SD=0.92] Lecturers with 11+ years experience agreed more strongly than those with less than 2 years experience. • In order to introduce innovative assessment I would consider avoiding official procedures. 27% agree57% disagree16% uncertain [Mean=2.64,SD=1.09] No effect for teaching qualification or length of experience ISSOTL 2009

15. What do new lecturers feel are external constraints? • Changes to my assessment design are sometimes hindered by external factors (e.g. cost, high student numbers, time). 75% agree20% disagree4% uncertain [Mean=3.81,SD=1.11] • QAA requirements seldom allow leeway in assessment design. 29% agree 41% disagree31% uncertain [Mean=2.91,SD=0.95] • If external constraints (e.g. QAA, subject benchmarks) were removed, I would be more willing to change my assessment practice. 35% agree41% disagree24% uncertain [Mean=2.98,SD=1.10] No effect for teaching qualification or length of experience for these 3 items ISSOTL 2009

16. What do new lecturers think about students when designing assessment? (1) • It is possible for students to ‘go through the motions’ to satisfy assessment requirements without learning anything. 57% agree39% disagree5% uncertain [Mean=3.22,SD=1.25] No effect for teaching qualification or length of experience • It is a challenge to design assessments for students of differing levels of commitment. 77% agree20% disagree4% uncertain [Mean=3.78,SD=1.09] No effect for teaching qualification or length of experience • I find it a challenge to design pedagogically sound assessment because of student focus on grades. 57% agree35% disagree10% uncertain [Mean=3.27,SD=1.09] Lecturers with 11+ years experience found this less of a challenge than those with 5 years or less experience. ISSOTL 2009

17. What do new lecturers think about students when designing assessment? (2) • I find it a challenge to design assignments for students of differing levels of ability. 61% agree35% disagree5% uncertain [Mean=3.32,SD=1.09] Lecturers with 11+ years experience found this less of a challenge than those with less than 2 years experience. ISSOTL 2009

18. Conclusions We are still at the early stages in our analysis but our interim findings suggest that: • Lecturers (both inexperienced and experienced) taking a university teaching course do think that it has changed their views on assessment. • Over three quarters of our respondents thought that assessment needs improving • The fact that there were some significant differences in responses between the experienced lecturers and those with less experience suggest that the former may be more likely to find their way around the various constraints when designing their assessments ISSOTL 2009

19. Further research • Further analysis of our findings from this study is being carried out and will inform our other work on developing a marking and feedback inventory (currently under construction) • The ADI itself is still under development and will be made freely available as a research tool for all on the Write Now website http://www.writenow.ac.uk • If colleagues would like to use the current version, they are welcome to do so but we would ask that they contact us first at cetlresearchteam@hope.ac.uk ISSOTL 2009

20. References • Becher, T. & Trowler, P. (2001) Academic tribes and territories: Intellectual Enquiry and the Culture of Disciplines 2nd edition Buckingham: Society for Research in Higher Education and the Open University Press. • Black, P. and Wiliam, D. (1998) Assessment and Classroom Learning.Assessment in Education 5 (1) 7-74 • Boud, D & Falchikov,N. (2007) Rethinking assessemnt in higher education. Learning for the longer term, Routledge • CETL: Assessment for learning http://www.northumbria.ac.uk/cetl_afl/whatis/ • Fanghanel, J. (2007) ‘Investigating university lecturers’ pedagogical constructs in the working context’. York: Higher Education Academy. http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/fanghanel.pdf • Norton, L., Aiyegbayo, O, Harrington, K., Elander, J. and Reddy, P. (in press) New lecturers’ beliefs about learning, teaching and assessment in higher education: The role of the PGCLTHE programme. Innovations in Education and Teaching International • Rust, C (2007) "Towards a scholarship of assessment" Assessment & Evaluation in Higher Education 32, 2, 229-237 ISSOTL 2009