1 / 66

What Can We Learn from Quantitative Data in Statistics Education Research?

What Can We Learn from Quantitative Data in Statistics Education Research?. Sterling Hilton Brigham Young University Andy Zieffler University of Minnesota John Holcomb Cleveland State University Marsha Lovett Carnegie Mellon University. Introduction. Components of a research program

tavia
Download Presentation

What Can We Learn from Quantitative Data in Statistics Education Research?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. What Can We Learn from Quantitative Data in Statistics Education Research? Sterling Hilton Brigham Young University Andy Zieffler University of Minnesota John Holcomb Cleveland State University Marsha Lovett Carnegie Mellon University University of Minnesota Educational Psychology

  2. Introduction • Components of a research program • Generate ideas (pre-clinical) • Develop a conceptual framework • Frame question (pre-clinical, Phase I) • Constructs and Measurement • Design and Methods • Pilot study • Examine question (Phase I, Phase II) • Establish efficacy (small) • Generalize findings (Phase III) • Larger studies in varied settings • Extend findings (Phase IV) • Longitudinal studies • Different populations

  3. Introduction • Quantitative methods in research program • Framing: measurement development • Validity and reliability • Framing: pilot study • Examine • Generalize • Extend • Statistics education research is primarily in the “generate” and “frame” phases

  4. Introduction • Purpose: Introduce two instruments that are in different stages of development and discuss how they have been and might be used in statistics education research • Comprehensive Assessment of Outcomes in a Fist Statistics course (CAOS) • Survey of Attitudes Toward Statistics (SATS)

  5. Assessment Resource Tools for Improving Statistical Thinking • Several online assessments • ARTIST Topic Scales • Comprehensive Assessment of Outcomes in a First Statistics course (CAOS) • Statistics Thinking and Reasoning Test (START)

  6. ARTIST Topic Scales • 7-15 MC items • Many topics • Data Collection • Data Representation • Measures of Center • Measures of Spread • Normal Distribution • Probability • Bivariate Quantitative Data • Bivariate Categorical Data • Sampling Distributions • Confidence Intervals • Significance Tests

  7. CAOS Test • 40 MC items • Designed to assess students’ statistical reasoning after any first course in statistics. • CAOS test focuses on statistical literacy and conceptual understanding, with a focus on reasoning about variability. • Developed through a three-year process of acquiring and writing items, testing and revising items, and gathering evidence of reliability and validity.

  8. CAOS Test • Reliability Analysis • Sample of 10287 • Cronbach’s alpha coefficient of .77 • Content Validity Evidence • 18 expert raters • Unanimous agreement that CAOS measures important basic learning outcomes • All raters agreed with the statement “CAOS measures outcomes for which I would be disappointed if they were not achieved by students who succeed in my statistics courses.” • Some raters indicated topics that they felt were missing from the scale - no agreement among these raters about the topics that were missing.

  9. START Test • 14 MC items • Identified through a principal components analysis performed on CAOS data gathered in Fall 2005 and Spring 2006 (n = 1470). • Alpha Coefficient from that data set was calculated to be 0.74.

  10. Use of Quantitative Measures in a Phase 1 Study • Exploratory Studies • What can we find out about students’ understanding? • Where are students having difficulties? • Are there inconsistencies in students’ reasoning?

  11. Example Item 1 Measured Learning Outcome Understanding the interpretation of a median in the context of boxplots.

  12. Example Item 1 The two boxplots below display final exam scores for all students in two different sections of the same course

  13. Example Item 1 Which section has a greater percentage of students with scores at or above 80? • Section A • Section B • Both sections are about equal.

  14. Example Item 1 Which section has a greater percentage of students with scores at or above 80? • Section A • Section B • Both sections are about equal.

  15. Example Item 1 • How did students answer this item?

  16. Example Item 1

  17. Example Item 1 • Is this surprising? • What can we learn from students’ responses to this item? • Implications/Directions for research? Teaching?

  18. Example Item 2 Measured Learning Outcome Understanding that correlation does not imply causation.

  19. Example Item 2 Researchers surveyed 1,000 randomly selected adults in the U.S. A statistically significant, strong positive correlation was found between income level and the number of containers of recycling they typically collect in a week. Please select the best interpretation of this result.

  20. Example Item 2 • We can not conclude whether earning more money causes more recycling among U.S. adults because this type of design does not allow us to infer causation. • This sample is too small to draw any conclusions about the relationship between income level and amount of recycling for adults in the U.S. • This result indicates that earning more money influences people to recycle more than people who earn less money.

  21. Example Item 2 • We can not conclude whether earning more money causes more recycling among U.S. adults because this type of design does not allow us to infer causation. • This sample is too small to draw any conclusions about the relationship between income level and amount of recycling for adults in the U.S. • This result indicates that earning more money influences people to recycle more than people who earn less money.

  22. Example Item 2 • How did students answer this item?

  23. Example Item 2

  24. Example Item 2 • Is this surprising? • What can we learn from students’ responses to this item? • Implications/Directions for research? Teaching?

  25. Example Item 3 Measured Learning Outcome Ability to match a scatterplot to a verbal description of a bivariate relationship.

  26. Example Item 3 Bone density is typically measured as a standardized score with a mean of 0 and a standard deviation of 1. Lower scores correspond to lower bone density. Which of the following graphs shows that as women grow older they tend to have lower bone density?

  27. Example Item 3 • Graph A • Graph B • Graph C

  28. Example Item 3 • How did students answer this item?

  29. Example Item 3

  30. Example Item 3 • Is this surprising? • What can we learn from students’ responses to this item? • Implications/Directions for research? Teaching?

  31. Example Item 4 Measured Learning Outcome Understanding of the purpose of randomization in an experiment.

  32. Example Item 4 A recent research study randomly divided participants into groups who were given different levels of Vitamin E to take daily. One group received only a placebo pill. The research study followed the participants for eight years to see how many developed a particular type of cancer during that time period. Which of the following responses gives the best explanation as to the purpose of randomization in this study?

  33. Example Item 4 • To increase the accuracy of the research results. • To ensure that all potential cancer patients had an equal chance of being selected for the study. • To reduce the amount of sampling error. • To produce treatment groups with similar characteristics. • To prevent skewness in the results.

  34. Example Item 4 • To increase the accuracy of the research results. • To ensure that all potential cancer patients had an equal chance of being selected for the study. • To reduce the amount of sampling error. • To produce treatment groups with similar characteristics. • To prevent skewness in the results.

  35. Example Item 4 • How did students answer this item?

  36. Example Item 4

  37. Example Item 4 • Is this surprising? • What can we learn from students’ responses to this item? • Implications/Directions for research? Teaching?

  38. How Can We Use the Results? • Begin to look for underlying reasons students are having difficulties • Examine the research literature • Interview students to gain a more in-depth understanding of their reasoning • Compare results with data from other classes (other teachers, schools)

  39. How Can We Use the Results? • They can inform our instruction • Reconsider how difficult or easy some concepts are for students • Rethink how we currently teach these ideas • Add new activities or tools • Re-allocate classroom time • Change the way we assess students • Assessment items better aligned with learning outcomes • Assessment items that probe students reasoning

  40. SATS • Survey of Attitudes Towards Statistics • Candace Schau and Tom Dauphinee (http://www.unm.edu/~cschau/satshomepage.htm) • Twenty-eight item survey • Seven point Likert scale response Strongly Neither agree Strongly Disagreenor disagree Agree 1 2 3 4 5 6 7

  41. SATS • Original four subscales • Value (9 items; α range .80 - .90 ) “Statistics is worthless.” • Affect (6 items; α range .80 - .85) “I like statistics.” • Cognitive Competence (6 items; α range .77 - .85) “I have no idea of what’s going on in statistics.” • Difficulty (7 items; α range .64 - .79) “Statistics is a complicated subject.”

  42. SATS • Two additional subscales • Interest (4 items) “I am interested in using statistics.” • Effort (4 items) “I plan to complete all of my statistics assignments.”

  43. SATS • Attitude is multi-faceted outcome • Issues to consider • Pre-existing attitudes • Direction and magnitude of changes over a semester • Relevance of items to study

  44. Using the SATS: A Case Study Assessment of a project-rich introductory statistics course • Fall 2004, at Cleveland State University • Class 1: 30 students Pre/Post • Class 2: 16 students Pre/Post • SATS administered first day and final exam day

  45. Class 1: Projects - Rich • 4 team projects that used/required • Real data • Computer Software • Collaboration • Writing • Individualized Mid-Term and Take-home Data Analysis Exams • http://academic.csuohio.edu/holcombj/eku/index.html • Login: holcomb pwd: projects22

  46. Class 2 • Ti – 83 • In – Class demos • Homework and Exams

  47. Comparison of Pre Data • No significant difference between Class1 and Class2

More Related