Mining NSS data for enhancement An example from Art & Design

1. Mining NSS data for enhancementAn example from Art & Design Mantz Yorke mantzyorke@mantzyorke.plus.com Higher Education Academy Conference “Higher Education Surveys for Enhancement” Nottingham, 18 May 2010

2. This presentation illustrates one way in which NSS data have been used in the interests of enhancement It ought to be adaptable to other situations

3. The paradox of Art & Design Art & Design makes a substantial contribution to the national economy. It is a national success story. Yet when it is viewed through the lens of the NSS, it does not come out well in comparison with other subject areas (save in respect of Assessment & Feedback). Question 1: Why? 

4. Variation in the provision of Art & Design Judged by NSS scores, some institutions do much better than others. Question 2: Why? 

5. Clearing some (substantial) brushwood I am critical of the NSS, on various conceptual and technical grounds. It is problematic as a basis for informing the public. It is however more useful in its ancillary role as a prompt to enhancing the student experience. • Even if it’s a gross ‘measure’, we can still get messages from it. Institutions are ill-advised to ‘chase the ratings’: rather, they need to look underneath the scores to identify points at which enhancement-oriented activity might most profitably be targeted.

6. Politics The NSS is part of a political agenda: it is here to stay (for the time being, at least – but the focus of interest is beginning to switch to student engagement, following the use of NSSE in the US and AUSSE in Australia). So we have to live with it, even if we don’t like it.

7. Art & Design and the NSS

8. Selected subject comparisons, NSS 2009 (New universities and colleges in England and Wales) Specialist equipment Explaining Interesting Enthusiastic Stimulating Timetable Changes Organisation Prompt Detailed Clarified NB Means are unweighted Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS

9. Selected subject comparisons, NSS 2009 (New universities and colleges in England and Wales) Specialist equipment Explaining Interesting Enthusiastic Stimulating Timetable Changes Organisation Prompt Detailed Clarified NB Means are unweighted Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS

10. The NSS and Art & Design – a case study GLAD, the HE Academy ADM Subject Centre and the HEAD Trust combined in supporting a study of the reasons underlying the relatively poor performance of Art & Design since the inception of the NSS. The study had quantitative and qualitative aspects: only the former is represented in this presentation. (See the HEA-ADM Subject Centre website for the report of the study.)

11. Disaggregating to institutions with studio-based provision Design Studies is used as an illustration: similar results are found for Fine Art and Cine & Photo Studies

12. Specialist A&D institution • Some institutions do (consistently) better than others • No ‘specialist institution effect’ • No particular size effect

13. Digging deeper into the NSS data Three examples

14. Design Studies, selected comparisons (NSS 2009) Explaining Interesting Enthusiastic Stimulating Specialist equipment Timetable Changes Organisation Novel problems Prompt Detailed Clarified Clear criteria Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS

15. Fine Art: selected comparisons (NSS 2009) Explaining Interesting Enthusiastic Stimulating Timetable Changes Organisation Specialist equipment Novel problems Prompt Detailed Clarified Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS

16. English: selected comparisons (NSS 2009) Specialist equipment Explaining Interesting Enthusiastic Stimulating Timetable Changes Organisation Novel problems Prompt Detailed Clarified Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS

17. Some possible determinants of the NSS outcomes Specific institutional and/or departmental causes (e.g. relocation, staffing difficulties). Programme leadership and/or management. Pedagogic approach in Art & Design. Curricular positioning of Art & Design in (some) multi-disciplinary institutions. Balance of FT and PT staff. Student expectations v. experiences: ‘The Deal’. A&D students having to be more self-organising than many others as regards booking spaces and resources; self-organising may cut across formal timetabling.

18. Some methodological points The illustrations are based on raw data. ‘Residuals’ make allowances for various institutional factors, such as entry profile, but their interpretation and presentation to others require statistical sophistication. The raw data may be rougher than one would like, because they are affected by local considerations and the limits of the NSS as an instrument. One year’s NSS scores may therefore be unreliable. Nevertheless, rough and ready data may be ‘good enough’ to act as prompts to enhancement. Warning Shaping the data into a usable form is a bit of a slog.

19. In conclusion • NSS data show relative strengths and weaknesses. • The ‘quality literature’ asserts that there’s almost • always something that can be improved, by • doing things better • or • doing things differently. • The important thing is to identify those aspects of • provision where enhancement activity is most likely • to pay off, and to focus on them. • There are unlikely to be many quick fixes.