Non-experimental Research

1. Non-experimental Research Research Process and Design Spring 2006 Class #8

2. Today’s objectives • To answer any questions you have • To review general comments about article critiques • To give/receive feedback on literature review outlines • To explore non-experimental research designs • To talk about writing the methods section • To distribute and discuss mid-term exam Research Process and Design (Umbach)

3. Feedback on literature review outline • Pair with another group and review their outline • How is it organized? Do they organize it around themes and ideas? • Based on the outline, can you see the argument they are making? • What suggestions do you have regarding structure? Resources Don’t be afraid to ask questions to get more details about how they are reviewing the literature. Research Process and Design (Umbach)

4. Non-experimental Research • Purpose • Describes current characteristics of interest or examines the relationships between things without any direct manipulation of conditions that are experienced by the subjects Research Process and Design (Umbach)

5. Non-experimental research designs • Descriptive • Comparative • Correlational • Ex post facto Research Process and Design (Umbach)

6. Descriptive Research • Purpose: to describe current status so as to provide a basis for asking additional questions • How much do college students exercise? • What are the attitudes of teachers toward mainstreamed children? Research Process and Design (Umbach)

7. Longitudinal Studies • Those in which the same group of subjects is followed over time • What changes have occurred in the GPAs of the senior class since entering the ninth grade? • Trend • Cohort • Panel • Limitations • Takes a long time to complete • Difficult to keep track of subjects and maintain their cooperation Research Process and Design (Umbach)

8. Cross-Sectional Studies • Different groups of subjects studied at the same time • What are the opinions of 7th, 8th, 9th, 10th, and 11th grade students toward an extended school year? • Advantage • Don’t have to wait years to complete research • Disadvantage • Selection differences between groups may bias results Research Process and Design (Umbach)

9. Comparative Research • Purpose: to investigate the relationship between variables by comparing the differences between two or more groups on those variables so as to provide a basis for identifying • 1) Possible causes of important outcomes • 2) Variables that need further investigation • 3) Predictor variables of importance Research Process and Design (Umbach)

10. Comparative Research • Examples • Is there a difference in the attitudes of male and female students? • What is the difference in the achievement levels of ninth grade students attending public, private, and parochial schools? • Are experienced teachers more effective than novice teachers? Research Process and Design (Umbach)

11. Comparative Research • Limitation: conclusions about causality are not warranted • Sampling and instrumentation are critically important to this type of research as are the procedures that are followed • e.g., the lack of manipulation of an independent variable Research Process and Design (Umbach)

12. Correlational Research • Purpose: to investigate the relationships between two or more variables by analyzing the correlations between and among them so as to provide a basis for identifying • 1) Possible causes of important outcomes • 2) Relationships that appear to need further investigation • 3) Predictor variables of importance Research Process and Design (Umbach)

13. Correlational Research • Types • Bivariate (correlation) • Prediction (regression) • Path Analysis • Logistic Regression • Structure Equation Modeling Research Process and Design (Umbach)

14. Bivariate • The relationships between two variables (i.e., one predictor and one criterion variable) • Examples • What is the relationship between math attitude and math achievement? • What is the relationship between socio-economic status (SES) and achievement? Research Process and Design (Umbach)

15. Bivariate • Use of multiple variables for which all bivariate correlations are examined • What are the correlations between science, math, English, social studies, and PE grades? • Use of bivariate correlation coefficients (e.g., Pearson Product Moment, Spearman Rho, etc.) Research Process and Design (Umbach)

16. Prediction • Prediction of a criterion variable from one or more predictor variables • Examples • How well do high school GPAs and SAT scores predict freshman GPAs? • How well do Praxis Examination scores predict teacher effectiveness? Research Process and Design (Umbach)

17. Prediction • Use of multiple regression coefficients • Multiple R • multi-variate correlation coefficient Research Process and Design (Umbach)

18. Prediction • Regression coefficients • Weights assigned to each predictor variable so as to be able to statistically predict the criterion variable • Weights that are in the metrics used by each variable • Beta weights: standardized coefficients that allow for the relative comparison of the effectiveness of each predictor variable to the predictive process Research Process and Design (Umbach)

19. Prediction • Use of multiple regression for statistical control by pulling the effect of an independent variable out of a dependent variable • Adjusting achievement scores for the effects of differing levels of student SES before comparing the performance of schools • Adjusting teacher effectiveness scores for the effects of experience prior to comparing teacher effectiveness across schools Research Process and Design (Umbach)

20. Prediction • Factors positively affecting prediction • Reliable measures • Shorter lengths of time between collection of predictor and criterion variables • Simple, uncomplicated criterion variables Research Process and Design (Umbach)

21. Interpreting Correlation Coefficients • Correlation does not imply causation • No manipulation of an independent variable • No controls for extraneous variables • Questions as to the direction of the relationship • Does attitude predict achievement? • Does achievement predict attitude? Research Process and Design (Umbach)

22. Interpreting Correlation Coefficients • Usefulness • Correlations less than .35 indicate little relationship and are of some use for exploratory purposes but of little use for predictive purposes • Correlations between .35 and .75 are useful as crude estimates of individual and group predictions • Correlations greater than .75 are useful for both group and individual predictions Research Process and Design (Umbach)

23. Interpreting Correlation Coefficients • Spurious correlations: correlations that over-represent or under-represent the actual relationship • Over-representation is often the result of a common variable that related highly to the predictor and criterion variables • Under-representation is often the result of unreliable measurement, restricted variability in scores, homogeneous samples, the inclusion of outlier scores Research Process and Design (Umbach)

24. Interpreting Correlation Coefficients • Size of correlation coefficients • Coefficient of determination • Squared correlation coefficient (i.e., r2 and R2) • Represents the proportion of variation in the criterion variable predicted by the predictor variable(s) • The closer the value to 1.0 (i.e., 100%) the “better” the magnitude of the correlation Research Process and Design (Umbach)

25. Interpreting Correlation Coefficients • Statistical significance • Correlation coefficients are often reported as significant (i.e., statistically different from 0) or non-significant (i.e., statistically equal to 0) • Significant correlations with large samples should be interpreted very carefully Research Process and Design (Umbach)

26. Ex Post Facto Research • Purpose: investigate whether one or more preexisting conditions have possibly caused subsequent differences in groups of subjects • What are the differences between students who drop out of school and those who stay in school? • What are the reasons for the high attrition rates for teachers in this district? Research Process and Design (Umbach)

27. Ex Post Facto Research Characteristics • Conditions have already occurred (e.g., students have dropped out of school, teachers have left the teaching profession, etc.) • There is a lack of ability to manipulate or control variables for moral, ethical, or legal reasons (e.g., the research cannot force a student to drop out of school or force a teacher to quit) Research Process and Design (Umbach)

28. Conducting Ex Post Facto Research • Formulate a problem that includes possible causes of the dependent variable • Identify rival hypotheses that might explain the relationships • Find or select the groups to be compared • Collect and analyze data on the factors related to the hypotheses and rival hypotheses • Draw conclusions about the original question(s) being sure to infer causes and effects very, very cautiously Research Process and Design (Umbach)

29. Group project • What design will you employ? • Who do you plan to study? • What and how will you collect data? • What are your independent and dependent variables? Research Process and Design (Umbach)

30. For next week… • Sampling • Readings: • *Light, Singer, & Willett – Ch. 3 • *Fowler – Ch. 1 Reminder: Mid-term due Research Process and Design (Umbach)