insights into interpreting the nss n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Insights into interpreting the NSS PowerPoint Presentation
Download Presentation
Insights into interpreting the NSS

Loading in 2 Seconds...

play fullscreen
1 / 33

Insights into interpreting the NSS - PowerPoint PPT Presentation


  • 66 Views
  • Uploaded on

Insights into interpreting the NSS. Mark Langan, Alan Fielding & Peter Dunleavy Manchester Metropolitan University Faculty of Science & Engineering

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Insights into interpreting the NSS' - jonah-henson


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
insights into interpreting the nss

Insights into interpreting the NSS

Mark Langan, Alan Fielding & Peter Dunleavy

Manchester Metropolitan University

Faculty of Science & Engineering

Fielding, A.F., P.J.Dunleavy and A.M. Langan (2010) Effective use of the UK's National Student (Satisfaction) Survey (NSS) data in science and engineering subjects. Journal of Further and Higher Education, 33, 347-368.

GEES NSS Workshop 2010

what makes a student satisfied

What makes a student satisfied?

GEES NSS Workshop 2010

structure

Research evidence based on a range of quantitative approaches using national dataset (science subject focus)

  • Discussion about implications of using NSS for decision-making in H.E.

Structure

GEES NSS Workshop 2010

slide4

Based on Course Experience Questionnaire and considered robust in terms of the statistical measures; internal consistency, construct validity and concurrent validity.

Measures six dimensions: teaching; assessment and feedback (sometimes considered separately); academic support; organisation and management; resources; and, personal development.

GEES NSS Workshop 2010

slide5

Five point scale with 21 ‘items’ and separate overall satisfaction rating (Q22)

Thorough overview can be found in Surridge 2007 and Marsh and Cheng 2008. Take home message: the outputs are hierarchical in nature and not designed for simplistic league tables

Note: satisfaction is a complex concept to measure and there are many approaches

GEES NSS Workshop 2010

slide6
Data

Level 3 (closest to Programme/Dept)NSS data from

  • 2007;
  • 2008;
  • 2009.

Science subjects data pruned to remove subjects not taught at MMU (e.g. medicine).

Still very large data sets (>40,000 cases per survey)

GEES NSS Workshop 2010

nss questions
NSS Questions

Teaching (Teach)

Q1 Staff are good at explaining things.

Q2 Staff have made the subject interesting.

Q3 Staff are enthusiastic about what they are teaching.

Q4 The course is intellectually stimulating.

Assessment fairness (Fairness)

Q5 The criteria used in marking have been clear in advance.

Q6 Assessment arrangements and marking have been fair.

Assessment feedback (Feedback)

Q7 Feedback on my work has been prompt.

Q8 I have received detailed comments on my work.

Q9 Feedback has helped me clarify things I did not understand.

nss questions1
NSS Questions

Support (Support)

Q10 I have received sufficient advice and support with my studies.

Q11 I have been able to contact staff when I needed to.

Q12 Good advice was available when I needed to make study

choices.

Management & Organisation (Org)

Q13 The timetable works efficiently as far as my activities are

concerned.

Q14 Any changes in the course or teaching have been

communicated effectively.

Q15 The course is well organised and is running smoothly.

nss questions2
NSS Questions

Learning Resources (Resources)

Q16 The library resources and services are good enough for my needs.

Q17 I have been able to access general IT resources when I needed to.

Q18 I have been able to access specialised equipment, facilities or room when I needed to.

Personal Development (PD)

Q19 The course has helped me present myself with confidence.

Q20 My communication skills have improved.

Q21 As a result of the course, I feel confident in tackling unfamiliar problems.

Overall satisfaction (Overall)

Q22 Overall, I am satisfied with the quality of the course.

satisfaction

Satisfaction is % of students answering 4 or 5 to a question.

e.g. Q 1 Biology MMU – 95% of students were satisfied

Note that dissatisfaction may tell a different story.

Satisfaction

GEES NSS Workshop 2010

subject differences general national
Subject differences (general/national)

Subject differences confound simple comparisons, examples from 2009.

Medians for Qs 7, 8 & 9 plus 13, 14 & 15.

GEES NSS Workshop 2010

what answers are correlated with q22
What answers are correlated with Q22?

Approach

  • Use % in agreement with a question (answers 4 & 5 on 5 point scale)
  • Simple correlation (ignoring subject)
  • Correlation allowing for subject differences (ANCOVA)
  • Repeat for each year
  • Calculated for national dataset (three years)

GEES NSS Workshop 2010

annual national trends ranks
Annual national trends (ranks)

Overall satisfaction is consistently related to:

Teaching Quality, Support and Organisation.

It only weakly related to Resources and Assessment, particularly feedback.

GEES NSS Workshop 2010

subject differences feedback qs
Subject differences (Feedback Qs)

GEES NSS Workshop 2010

predictive model forest tree analysis

Decision and regression trees use a supervised learning algorithm which must be provided with a training set that contains cases with class labels or values for classification.

  • We used a new variant of regression trees called FTA (software called ‘RandomForests’). Robust method with fewer constraints than traditional regression methods, for example allowing different factors to be explored in their influence on overall satisfaction within different subgroups.

Predictive model (Forest Tree Analysis)

GEES NSS Workshop 2010

regression trees an example based on http www dtreg com classregress htm predicts property value
Regression Trees (an example)based on http://www.dtreg.com/classregress.htmPredicts property value
slide22

Top five predictors (best first)

Q15 The course is well organised and is running smoothly.

Q4 The course is intellectually stimulating.

Q1 Staff are good at explaining things.

Q14. Any changes to the course or teaching have been communicated effectively.

Q10 I have received sufficient advice and support with my studies.

GEES NSS Workshop 2010

conclusions

Subject differences (e.g. mathematical content)

  • Institutional differences
  • False assumptions (e.g. enhancing feedback directly enhances Q22)
  • Institutional effects
  • Satisfaction is a complex measure related to L&T practices
  • Teaching and Organisation prominent

Conclusions

GEES NSS Workshop 2010

quotes from ramsden 2007

“… [The NSS] is not a measure of satisfaction so much as a window into how our designs for learning are experienced by students.

From these insights we assemble the

practical measures we may take to

enhance the quality of their

experiences.”

Quotes from Ramsden (2007)

Higher Education Academy Annual Conference 2010

quotes from ramsden 20071

“... it is not simple to know what to do. Current experiences, unlike satisfaction, are a mixture of previous experiences and the environment as it is now...

... so sometimes we will need to adjust expectations or consider altering previous experiences in order to improve quality.”

Quotes from Ramsden (2007)

Higher Education Academy Annual Conference 2010

quotes from ramsden 20072

“I cannot agree with the idea, for example, that because students are slightly less positive about feedback on assessed work in the NSS than about the quality of teaching...

... we should rush to bully academics into providing more feedback more quickly.”

Quotes from Ramsden (2007)

Higher Education Academy Annual Conference 2010

quotes from ramsden 20073

“From this it also follows that students do not have a ‘right’ to be satisfied. They are themselves part of the experience. ..

... Students decide their own destinies and we can only add or subtract value at the margins.”

Quotes from Ramsden (2007)

Higher Education Academy Annual Conference 2010