Loading in 2 Seconds...
Loading in 2 Seconds...
Using Multivariate Techniques for the Analysis of Survey Data: A Case Example. Isabelle Bourgeois Natural Sciences and Engineering Research Council University of Ottawa Joint Conference of the CES and the AEA Toronto, October 26 th 2005.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Using Multivariate Techniques for the Analysis of Survey Data: A Case Example Isabelle Bourgeois Natural Sciences and Engineering Research Council University of Ottawa Joint Conference of the CES and the AEA Toronto, October 26th 2005
Performance measurement activities in R&D often use complex techniques taken from the sciences and economics Methods are needed that capture more fully the non-economic benefits of research to support decision makers The use of multivariate analysis techniques taken from the social sciences can provide useful information based on survey results Rationale for Study
Industry awareness of government research and researchers Satisfaction with past interactions between industry and government Level of trust in researchers and staff Types of new processes or products introduced to market Examples of Survey Measures
Traditional survey data analysis relies on descriptive statistics with little exploration of the relationships that exist between variables Harman (2004) example: Use of t-tests with little interpretation and no further research Use of Sophisticated Analytical Methods
Multivariate analysis conducted on data from the Postgraduate Scholarship (PGS) program managed by NSERC Data collection instrument: survey administered in the summer of 2005 (N=901) Attrition: 101 Response rate: 69.6% (n=557) Survey results estimated to be accurate ±3%, 19 times out of 20 Case Example
Independent: gender, type of award received (master’s or doctoral), and main field of study Dependent: four seven-point scales on different aspects of the student’s experience with the PGS award Variables
Descriptive statistics to get a general sense of the survey results Verification of statistical assumptions most critical to multivariate analysis, normality and equality of the variance-covariance matrix (homoscedasticity) Preliminary Examination of Data
MANOVA (Multivariate Analysis of Variance): yields information to answer questions about the combined effect of the dependent variables MANOVA conducted on each of the four scales using the gender, award type and field variables (Tables included in the paper) Multivariate Respondent Profile
Field variable found significant on its own in scale A2 and in combination with other dependent variables in scales A2 and C3 Suggests that the field of study has an impact on a student’s undergraduate experiences (A2), as well as his or her graduate experiences (C3) Findings of MANOVA (1)
Award variable found significant on scale C2 which focused on relationship with supervisor and future career plans The award variable has an impact on the student’s relationship with supervisor and future career plans – Master’s and Ph.D. students have different experiences and expectations Findings of MANOVA (2)
Close relative of the MANOVA Instead of looking at the influence of the independent variables on survey responses, it predicts group membership within each independent variable by using the responses to the scale items. Findings focus on items loading most highly for each variable – these provide the best discrimination between levels of the variable (examples) Discriminant Function Analysis
Aside from providing more sophisticated information about the results of the survey, multivariate analysis methods also provide information about the properties of the instrument itself. Exploration of Scale Properties
Data summarization technique Identifies underlying, or latent, dimensions in a dataset that, when interpreted, describe the data in a smaller number of concepts The responses to the items on the PGS survey scales are hypothesized to vary as a function of one or several latent variables Exploratory Factor Analysis
Nine meaningful factors extracted and retained for interpretation Example (Factor 1, Learning and Skill Development): Funding from NSERC will help me to complete my degree faster Theoretical knowledge of the discipline Analytical techniques/experimental methods Use of laboratory equipment or instruments Project management Etc. Factors Extracted
Learning and Skill Development Supervision Resources Career Orientation No Debt Encouragement Collaborative Experiences Debt Load Job Prospects Factors Extracted
Reliability: Internal consistency of a scale – minimizes the contribution of random error to item scores Alpha coefficient is used to measure internal consistency and is interpreted as a correlation Alpha coefficients calculated for each of the scales and the combined items, as well as for the Factors extracted in the EFA to see which model fits best Scale Reliability
Original Scales Scale A2: 0.59 Scale C1: 0.81 Scale C2: 0.77 Scale C3: 0.85 Combined Items: 0.82 Factors Factor 1: 0.84 Factor 2: 0.91 Factor 3: 0.81 Factor 4: 0.64 Factor 6: 0.50 Factor 8: 0.42 Factor 9: 0.24 Reliability Comparison
Easy to do using standard statistical software packages Interpretation is fairly straightforward Adds to descriptive data by highlighting the influence of the combined dependent variables on the survey responses Allows for a review of the instrument properties and provides suggestions for improvement Advantages of Multivariate Analysis