Doctoral Examination

1. Doctoral Examination Margaret KileyCEDAMMargaret.kiley@anu.edu.au

2. Overview • The current situation regarding the examination of theses • Early research re the selection of examiners • How examiners examine theses • Discussion

3. Where did you do your research degree? What examination process did it involve?What was good about it and what was not? If you had an ‘Oral’ was it the European? the Scandinavian? the USA? the UK? the Canadian? the New Zealand? Or the …model?

4. Current situation in Australia • Fully external & written thesis only item of examination • Final written thesis sent to two (sometimes three) external examiners i.e. examiners external to the university • Approx 50% of all Australian theses sent overseas • Confidential (but not necessarily objective) Examiners names not disclosed to candidates • Examiners, without contacting co-examiners, prepare written report (av 3-4 pages) with formative feedback • Most universities allow for possible oral examination

5. System (cont) • Examiners provide recommendation to university e.g. ‘accept as is’, ‘accept with minor / major changes’, ‘revise & re-submit’ - these are not necessarily gradings! • University examiners’ board makes final decision based on recommendations • Time from submission to varies (depending on recommendation) but Mean of 6 months - with extremes

6. Selecting examiners - some problematic choices • Examiner chosen for expertise in different aspects of the thesis   • Cross disciplinary project where the examiners come from different disciplines • An experienced and an inexperienced examiner/supervisor • Examiners from different countries • Examiners from industry or research institutes

7. Selection criteria • Research track record, leader in the field • A fit between examiner expertise and thesis topic • Examining & supervisory experience • An understanding of an Australian PhD • High standards but fair • Intellectual courtesy and generosity • Not prejudiced against methodology or style • Not pedantic • Availability and reliability

8. Experienced Examiners report that they… • Expect the student to pass as they open the thesis • Are very reluctant to fail a student with most experiencing considerable distress if they do so • Come to a decision about the quality of a PhD by about the end of Chapter 2 • Have a formative rather than summative view of thesis examination • Believe that there is a risk attached to sending theses to inexperienced examiners • Are reluctant to take much notice of institutional criteria when examining

9. Experienced Examiners appear to… • Be fiercely independent in their views • Hold varying views about the purpose of the PhD. (Is it the thesis or the student being examined?) • Consider professional duty as the main reason for examining, followed by the fact that they are going to be needing examiners for their own students! • Devote considerable time to examining each thesis • Have surprisingly inclusive approaches to methodology/ paradigm • Demonstrate few discipline differences in their responses, other than regarding publications

10. Inexperienced Examiners • Report that they have a high level of confidence in their ability to examine (which is not always reflected in what they say in response to other questions) • Suggest experience comes from supervising & examining Honours students and theses and reviewing manuscripts • Adopt a similar approach to the actual process of examining as do their more experienced colleagues, although they are more likely to focus on the ‘steps’ or components of a PhD rather than the whole

11. Inexperienced Examiners … • At a surprisingly high rate, wanted to fail first thesis or said it was ‘awful’ • Follow institutional criteria more than experienced colleagues • Felt (some of them) that they were being examined too • Suggest that one of their main difficulties is their inability to benchmark • See their role as maintaining standards and performing their summative assessment role correctly

12. Is there a difference? • From work of Trafford (2003) from 130 vivas it was possible to determine that: • Experienced examiners tended to ask questions that can be defined as ‘Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept’ • Inexperienced examiners tended to ask more ‘technical’ questions • Trafford, V. (2003) Questions in doctoral vivas: Views from the inside, Quality Assurance in Education 11(2) pp 114-122

13. Trafford’s Categorisation Innovation and DevelopmentHIGH C. Questions generally related to issues such as research question, choice of topics, location of study D. Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept Scholarship & Interpretation A. Types of questions include resolving research problems, content, structure B. Implications, awareness of, and familiarity with wider literature LOW

14. Strategies for examining • Different examiners approach the task differently, but most: • Begin by reading the Abstract, Introduction & Conclusion to gauge the scope of the work and whether what candidates say they are going to do is actually done • Looking at the references to see what sources have been used and whether they need to follow up on any of them • Then read from cover to cover taking detailed notes, finally go back over the thesis to check whether their questions have been answered or whether their criticisms are justified

15. The reports demonstrate… • A less than ideal thesis has… • Too much detail with lack of analysis • Lack of confidence, energy & engagement by the candidate • Lack of argument and rigour • Shoddy presentation (typos etc) • Lack of critique of own analysis/ sweeping generalisations based on opinion rather than analysis • Inadequate or poorly expressed methodology & scope A ‘good’ thesis has… • Critical analysis & argument • Confidence & a rigorous, self-critical approach • A contribution to knowledge • Originality, creativity & a degree of risk taking • Comprehensiveness & scholarly approach • Sound presentation & structure • Sound methodology

16. The Doctoral curriculum • Changes to the nature of research • Researchers are more mobile (literally through ease of travel and through the use of technology) • The knowledge society requires increasing numbers who have highly professional knowledge & skills • Time limits on doctoral scholarships/programs demand re-thinking of the process of ‘getting a doctorate’ • Purpose of the Doctorate • To make an original contribution to knowledge, and • Demonstrate the achievement of a range of relevant research skills • How to assess both?

17. Assessment for learning • Substantial evidence that one of the most significant, yet generally unrecognised, outcomes of the current examination process for Australian theses is the formative aspects of the written comments provided by examiners • This highly valuable feedback comes at the end of the Doctoral process, often too late to be incorporated into the research • Formative assessment is different from the summative “tick-a-box” recommendation that examiners are asked to make • Is there some more effective way of incorporating feedback?

18. Key questions • What is the purpose(s) of the Australian Doctorate? • Do the assessment mechanisms adequately reflect purposes? • What is the object of assessment? • Who assesses? • What form should the assessment take? • Do we need to have only model for all: • Disciplines? • Universities? • Can the assessment be a more meaningful and useful learning experience?

19. The ‘object’ of assessment • The student as a trainee researcher? • The dissertation as a contribution to knowledge? • With the increase in courses in the Doctorate should we be formally assessing? Why? How? When? e.g. coursework (20%); research proposal (10%); research thesis (70%)? • Should we assess students’ general knowledge of the discipline as well as knowledge of the specific research topic?

20. Who assesses? • What are the advantages and disadvantages for students and their learning of the University making the decision on the outcome of the thesis rather than the examiners? • What is the value to the student, university, and/or supervisor of having internal as well as external examiners? Should views of one carry more weight than the other? Why? • What are the advantages and disadvantages of bringing one or more external assessors to the University for a face-to-face oral, either during candidature or at the end?

21. Form of assessment: Possible ‘Oral’ • In many Australian universities orals are viewed negatively by students as they link oral with problems with the thesis • However, well handled orals can become ‘opportunities to fine-tune the written work of the candidate in order that it reaches a notional standard that is acceptable for scrutiny by peers in the relevant intellectual community’ (Green, 2005) • Oral exams also offer opportunities for inexperienced assessors to learn from more experienced colleagues • There are many different orals and so we would need to develop an Australian oral to suit the Australian Doctoral experience

22. Purposes of an ‘Oral’? • ‘Policing’: Did they actually do the work? • ‘Graduate attributes’: Oral communication skills • ‘Career/academic development’: knowledge of broad discipline • ‘Assessment’: Assess quality of the work & come to consensus • ‘Formative’: feedback to improve work • ‘Developmental’: Advice on future developments • ‘Research culture’: Public forum for research • Celebrate achievement

23. What form could the assessment take? • Could an oral assessment be an integral part of the assessment process: e.g."an internal approval to submit" or “public defence” as in Europe, or a relatively “private” discussion as in the UK? • What advantages and disadvantages are there of having a presentation by the student as a prerequisite for submission? For the student? Supervisor? University? Examiner? • If there is to be some form of ‘oral’ how public or private should it be?

24. Resources Australian Deans and Directors of Graduate Studies National Guidelines for Doctoral Thesis Examinationshttp://www.ddogs.edu.au/cgi-bin/index.pl Kiley, M. (2004). What examiners' comments can tell us about the postgraduate learning environment. In C. Rust (Ed.), Improving student learning: Theory, research and scholarship (pp. 213-222). Hinckley, Leicestershire: The Oxford Centre for Staff and Learning Development. Kiley, M., & Mullins, G. (2004). Examining the examiners: How inexperienced examiners approach the assessment of research theses. International Journal of Educational Research, 41(2), 121-135 Mullins, G. & Kiley, M. (2002) ‘It’s a PhD, not a Nobel Prize’: How experienced examiners assess research theses. Studies in Higher Education 27(4) pp. 370-386 SORTI web site at the University of Newcastle has information on examining theses, especially in the performing/visual arts http://www.newcastle.edu.au/centre/sorti/publications.html Trafford, V. (2003) Questions in doctoral vivas: Views from the inside, Quality Assurance in Education 11(2) pp 114-122