Conducting programme evaluations/reviews with a view to sustainable quality promotion

1. Conducting programme evaluations/reviews with a view to sustainable quality promotion Jan Botha University of Stellenbosch South Africa Keynote Address Oman Quality Network Conference Muscat, 28 October 2008

2. Key questions • Can we apply the 80/20 principle to programme evaluations? • What are universities learning from the evaluation of (formative) programmes?

3. Outline of the paper • Suggestions for the academic leader • QA in higher education context • Misconceptions about QA in higher education context • Programme evaluations and the 80/20 principle • What are we learning from programme evaluations

4. Suggestions for the academic leader

5. The predicament of the academic leader • My research … • I have become “one of them” • In the middle of the sandwich • From critic to custodian • I have to “manage”… “quality”

6. Academic Leadership Capabilities • “Learning Leaders in times of change” • Geoff Scott, Hamish Coates, Michelle Anderson (May 2008)

7. Four suggestions for the Academic Leader • Use quality management as instrument to achieve your vision and goals for your School / Department / Programme • Quality management can be a powerful change mechanism • Synchronise and integrate evaluation activities • Use an academic and research-informed approach to quality management

8. QA as an Academic Project • QA to be approached as a form of research • an intellectual enterprise • driven by curiosity, creativity and pragmatics • underpinned by social-scientific inquiry • accompanied by experimentation and reflection • informed by theorising • resulting in application and publication

9. A Principle of Design Specifications are continually moving upwards Opportunity for incremental changes Specifications Calculate this cost Poor design Poor design Good design

10. Quality Assurance in Higher Education Context

11. “Living with ideology in the university” • Ronald Barnett* The “idea of the university” as “a site of reason” • Universities are beset by many ideologies Foucault: ideology is “that which is opposed to truth” • Competition and Entrepreneurialism • Quality • Managerialism • Research • Access, inclusivity, multicultiralism • Turn the power of ideology onto itself, move from ideologies to ideaologies – recall the idea of the university * Beyond all reason. Living with ideology in the university. 2003. Open University Press.

12. The idea of the university?

13. What are universities for? • Universities until recently believed their existence and value was self-evident • Pressure of governments: universities should address the needs of society (and become “skills factories”?) • Universities state their vision, mission and objectives: • A place of quality, a place to grow • Creating futures • Your Knowledge Partner • Innovation Generation • To advance Learning and perpetuate it to Posterity • Sultan Qaboos University aspires to be an academic institution of excellence recognized both nationally and internationally, a center for learning where individuals can develop their capabilities in an environment that promotes academic achievement and research excellence; and a community where the highest moral and ethical values prevail for the purpose of both self-advancement and service to the community.

14. International snapshot • International Network for Quality Assurance Agencies in Higher Education (INQAAHE)… a world-wide association of some 200 organisations • Members include • Oman Accreditation Board • Commission for Academic Accreditation, UAE • Netherlands Quality Agency (NQA) • Malaysian Qualifications Agency (MQA) • Higher Education Commission, Pakistan • CONEAU - Comision Nacional de Evaluacion y Acreditacion Universitaria, Argentina • National Accreditation Agency (NAA), Russian Federation

15. India • The National Assessment and Accreditation Council (NAAC) • accredits institutions of higher education • size of Indian system: assessing hundreds of general education programs in 12 000 institutions of higher education is not feasible • institutional accreditation for general education institutions has been the strategy

16. United States of America • Many decades of institutional and programme accreditation in the USA • Council on Higher Accreditation Accreditation • Six regional accreditation organizations (MSA, SACS, NEASC, NWCCU, NCA, WASC) • Specialised accreditation agencies, e.g. ABET • 2,800 programs at more than 600 universities, over 1,500 dedicated volunteers participate annually in ABET activities • As many as 142 calendar days devoted in one year to visiting panels

17. South Africa • Council on Higher Education (CHE) • “just hang on, it will go away, as has happened with the other acronymous bodies…” • Institutional Audits • Programme Accreditation • National Reviews of Programmes • Quality Promotion and Capacity Building • Three steering instruments in Higher Education

18. (Slide: Ian Bunting)

19. Misconceptions about Quality Assurance in Higher Education

20. Challenging the “management” of quality • Is “managerialism” not contradictory to the nature of a university ? • “Management” of quality used to cover for poor quality? • More management of quality results in less quality… • CHE publication • Academic Freedom, Institutional Autonomy and Public Accountability in South African Higher Education (August 2008) (www.che.ac.za)

21. Misunderstandings (1) • Point of departure: lecturers are doing poor quality work, • we need QA to ‘catch the lazy buggers out” • I am satisfied with 95% of the work of the people in my university/faculty/institution • therefore QA is not necessary • Just make sure that you meet the minumim standards and “keep them off your back” • Aim for the minimum • Comply…

22. Misunderstandings (2) • If good academics are appointed at universities, we do not need QA systems. If the wrong people are appointed, no document (however thick it is), will ensure quality. • If nobody steals, we do not need laws and police. If there are thieves, no law (however thick it is), will ensure that no theft takes place. • If nobody is ill, we do not need doctors. Since we know beforehand that everyone is in any case going to die some day, the whole medical profession is senseless.

23. The case for Quality Management • Quality management is a necessary condition for quality, however, • quality management is not sufficient for quality

24. Programme evaluations and the 80/20 principle

25. Acknowledgement • Co-researchers • Prof Carools Reinecke • Emeritus Vice-Chancellor, Potchefstroom University (South Africa) • Mr Mr Kamal Bhagwandas Jogibhai • University of the Witwatersrand, Johannesburg, South Africa • Research Assistant • Mr André Müller, University of Stellenbosch

26. The Pareto Principle • Vilfred Pareto (1848-1923) • Wealth distribution in 19th Century England is predictably unbalanced • Most income and wealth go to a minority of people • Consistent mathematical relationship between proportion of people and amount of income and wealth • Principle of least effort • Principle of Imbalance • Zipf’s Principle of Least Effort • Resources tend to arrange themselves to minimize work • 20-30 % of resources accounted for 70-80% activity • Jurian’s Rule of the Vital Few or the 80/20 Principle

27. The 80/20 Principle illustrated Effort Results R Koch 2007. The 80/20 principle. The secret of achieving more with less. London: N Breadly Publishing

28. Purposes and types of programme evaluations • Mouton & Babbie(The practice of social research. Oxford, 2001): • judgement-orientated evaluations • improvement-orientated evaluations • knowledge-orientated evaluations • Trow(Academic reviews and the culture of excellence. Stockholm1994): • internal supportive • internal evaluative • external supportive • external evaluative • Evaluation outcomes used by different role players for different purposes

29. What is a programme? • A social intervention • A set of planned, structured learning experiences leading to a qualification • One qualification with various programmes leading to that qualification • BSc in Biodiversity and Ecology • BSc in Mathematical Sciences • One programme in a “field” with various qualifications • The “programme in mathematical sciences” consists of a Certificate, Diploma, Bachelor, Masters, and PhD

30. Unit of evaluation/review • An institution as a whole • An academic organisational units within an institution • department, centre, institute, bureau, school, faculty • Specific activities or processes within an institution • learning and teaching programmes • research programmes • community engagement programmes • capital campaigns

31. Types of programme evaluations • Evaluation of professional programmes as part of the periodic programme accreditation process • conducted by professional councils • National reviews of programmes • conducted by the a national QA body • the same programme evaluated and (re)accredited simulatenously at all institutions • Internal programme evaluations/reviews (for different purposes: improvement/closure/consolidation etc.) • conducted by institutions • professional and formative programmes • undergraduate and postgraduate programmes

32. Challenges in programme evaluation • Coherence within broader internal and external QA system • Formative programmes as unit of evaluation • BA, BSc, BComm, B SocSc • Spread over many departments in Schools and Faculties • Conflating “evaluation” with “reporting the results of an evaluation process” • Too much effort and resources for outcomes? 80/20 • Administrative burden • Evidence

34. Expectations of Programme Evaluations • Sustainable quality promotion • An instrument for change management • Integrated with other forms of evaluation • Outcomes should justify the effort • for academics and evaluators and HE managers • Standard methodology used • self-evaluation based on criteria/standards, report, panel visit, report, improvement action • Applicable to formative programmes

35. A “Pareto Approach” to Programme Evaluation • Consider the ideal situation • refer to criteria / standards • clustered in 11 themes • simplified (and “translated”) • Formulate a key question (or questions) • focus, contextualise (for us) • Express the evaluation in a symbol (5 point scale) discipline to make a judgement • Motivate the evaluation symbol (brief narrative, with substantiating evidence) • Formulate strategies (in bullet points)

36. Template for the Self-evaluation Report • Programme rationale Criteria The programme is consonant with the faculty’s mission, planning and resource allocation. The design maintains an appropriate balance of theoretical, practical and experiential knowledge and skills. It has sufficient disciplinary content and theoretical depth at the appropriate level.The programme offers opportunities for community interaction.The design offers learning and career pathways to students with opportunities for articulation with other programmes within and across institutions, where possible. The 2003 formulation of the strategic importance of the programme was re-evaluated. 1.2 Key Question 1.3 Evaluation 1.4 Motivation 1.5Strategy Reports limited (8,9 pages). Strategies formulated as actions

37. Criteria/standards clustered in themes • Programme rationale • Academic integrity • Student recruitment, admission and selection • Staff • Learning facilitation • Assessment • Infrastructure and academic information sources • Programme co-ordination • Student success and academic support for student success • Service Learning and work-based learning • Programme Evaluation and Development • Research basis (postgraduate programmes)

38. Case study • Faculty of Agri-Sciences, Stellenbosch University • BScAgric in Animal Production Systems; Agricultural Economics; Wine Production Systems; Crop Production Systems; Forestry; Food Science • (BAgric at Elsenburg College) • Faculty of Sciences, Stellenbosch University • BSc in Physics; Chemistry, Mathematical Sciences, Earth Science, Biodiversity, Molecular Biology, Human Life Sciences, Sport Science, Education • Faculty of Humanities, University of the Witwatersrand (Johannesburg)

39. Agri-Sciences

40. Agri-Sciences

41. Sciences

42. Sciences

44. What are we learning from programme evaluations?

45. 1. Programme rationale (69) • To enhance interaction with stakeholders(subject specific societies, industry, extraordinary lecturers, alumni) to broaden academic and industry specific networks (through a voluntary advisory committee) • To review/restructure the content/focusof the programme to ensure that module outcomes strongly reflect the programme specific and generic outcomes, reviewing programme structures annually and implementing student feedback and industry input, and developing a formal system to gather information from graduates on their appraisal of the course, and suggestions for improvement. • To adapt and better integrate courses, share expertise/reduce duplication and use resources more cost-effectively, reducing lecturing load by constructing generic courses across departments and including and designing modules to fill theoretical gaps and deepen theoretical knowledge or to add attractive career paths • To define service/experiential/work based learningwithin the programme and incorporate electives/optional subjects to better prepare students for the work environment, extending practical training/internship periods and awarding credits for lab or field work experience; To offer more opportunities for community interaction, especially for graduate students.

46. 1. Programme rationale (continued) • To outcomes communicate clearlyto contextualize lectures, and information about administrative and support services to students, staff and stakeholders (advertising student assistantships effectively) • To identify and support students at risk in small groups and give academic and non-academic help and to communicate the importance of class attendance; also addressing low/increased student numbers • To sharpen recruitment and marketing efforts, improving outreaches where scholars, prospective and current students can be informed about the programme and its strategic value. (Open days, information sessions...)

47. 2. Academic integrity (65) • To increase reseach and benchmarking opportunitiesfor staff with international scholars, to ensure the programme remains at forefront of new developments, and encouraging research/teaching interaction; • To review the undergraduate programmesannually, continuously and coherently planning modules in terms of content, level of difficulty and credit value; constantly reviewing the curriculum, learning materials, learning methods and programme outcomes; moderating it and ensuring that the feedback from external moderators is taken into account • To increase collaboration between lecturers to improve programme cohesion, filling in theoretical gaps through new and adapted modules, and cutting out duplication; also working with timetabling administrators to free up module combinations; allowing greater articulation with cognate programmes, structuring the programme to expose students to the core themes, balancing practice and theory to ensure the relevance of prescribed modules that are presented by other departments from both within, and external to, the faculty • To identify and remove obstacles inhibiting the use of experential learningand increase laboratory time, re-evaluating the module composition and structuring of the programme in order to make place for a longer period of internship; to evaluate the efficacy of the practical portions of the modules and investigate coherent year-long practical modules at second and third year levels; • To communicate the reason for the approach followed during the first year of study, and maintain a challenging learning environment for students despite low/increased numbers

48. 2. Academic integrity (continued) • To develop writing, information and computer skillsand improve the teaching of practical skills through the use of continuous assessment for all modules from second year upwards, and to identify and adapt modules fairly early in the academic programme; To focus on improving student mathematic skills; To complete computer literacy and scientific communication skills modules in the first year • To manage or establish new research institutes/units/centres such as an Institue for Food Technology (IFT) or Institute of Food Science and Technology (IFST) • To recruit students from historically disadvantaged backgrounds • To recruit more international students

49. 3. Student recruitment, admission and selection (93) • To involve industry stakeholders in recruitment and funding for undergraduate scholarships; To acquire additional bursaries from all sector role players; To expand facilities and staff; To increase support staff, especially in terms of popular media and marketing; To involve subject societies in recruitment actions; To increase interaction with bursary providers about the criteria for successful students and the problems that are experienced by current bursary holders; To investigate the roles professional bodies play in promoting the image of certain careers • To reviewadmission requirements for mathematics and science related to forest science programme; To evaluate the tempo at which graduates are employed; To investigate existing recruitment actions and material in terms of success and relevance; To monitor the extended degree programme to ascertain whether it is increasing student diversity; To monitor student numbers at the beginning of the year and consider the adjustment of entrance criteria if necessary; To review policy on ranking on application form so as to admit students who desire to follow; To review academic support available to struggling students in all years • To adapt intake in terms of quantity, quality and equity of students. To align recruitment efforts of the university, faculty, programme-committee and departments; To develop and implement mechanisms, including assessment methods, to widen access, (e.g. summer school, bridging programme); To increase the diversity of the student body in terms of South African cultural groups and international students; To create a structured marketing and recruitment plan as top priority, buying in expertise if needed, consulting with SU marketing divisions, utilizing existing marketing mechanisms, negotiating for undergraduate bursaries and addressing the student diversity; To improve the administrative implementation of admission criteria; To reconsider admission requirements at first year level to curb the high failure rate, as well as at honours level (possibly); To re-assess the modules which have become “service courses”;

50. 3. Recruitment, admission and selection (continued) • To make the bridging programme compulsory for studentswith a matric mark between 50%-56%; To visit underprivileged schools in our immediate vicinity; To sponsor prizes (e.g. book prizes) for the best Biology (now Life Sciences?) student in Grade 12 at a few selected schools; To consider an orientation period including visits to departments as well as the experimental farm before registration of second-year students to allow informed choices on major subjects; To supply information on programmes at the Expo for Young Scientists and Olympiad candidates, as well as high school science teachers; To encourage third years to attend final years’ product development presentations; To focus on the recruitment of coloured students due to the demand from Agri businesses; • To initiate discussions with the govrnment on fragmentation of higher forestry education in South Africa; To broaden the communication base with students at first- and second year level; To develop a culture of uniqueness and excellence amongst staff members and students; To raise awareness of language issues in the academic programme • To implement an extended degree programme(and first year academy) to benefit students that have to overcome academic inadequacies due to historical barriers, and help students to overcome cultural and language difficultiesTo prevent over-subscription to the course; Cap student numbers (no more laboratory space available);.; To identify the optimum number of students that can be accommodated in the programme (estimated at between 40 and 50 students), keeping economic factors in mind; To market a BSc(Hons) as a career route – both commercially and in parastatal, particularly academic spheres; To pay careful attention to entrance requirements in the next few years as the new school leaving certificate is implemented. Aim to raise entrance requirements; To critically review failures and examine the reasons for these