1 / 40

Data Collection, Publishing, and Dissemination of Results

Data Collection, Publishing, and Dissemination of Results. Eric Grodsky , University of Minnesota Chandra Muller, University of Texas at Austin. Motivation & Overview.

Download Presentation

Data Collection, Publishing, and Dissemination of Results

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Data Collection, Publishing, and Dissemination of Results Eric Grodsky, University of Minnesota Chandra Muller, University of Texas at Austin

  2. Motivation & Overview • The impact of the STEP program and your STEP Type 1 project depend on dissemination of knowledge for application to other settings. • Based on our early experiences working with STEP Type 1 projects • Steps to designing a study, collecting data, publishing and disseminating results: an overview and then example

  3. Study Design • A good study design makes it easier to publish and disseminate results • Begin the design phase early, alongside the program design • Consider collecting baseline (and control group) data before beginning program implementation

  4. Control or Comparison Group • What is the impact of your program on production of STEM majors? To answer this you must be able to estimate: • What would have happened without the program? • And ideally, what the program did to have the effect, and for whom? • Consider collecting a control group sample while you are planning the program, before it is implemented • Target students for comparison who are similar to those you will target for the program

  5. IRB: Institutional Review Board • Work with the IRB at your university to gain approval for your study • It is likely that your study will be considered “Exempt” • Example from University of Texas:

  6. Data Collection & Analysis • Study design and response rates are crucial • A randomized control trial—where the only difference between your control and treatment groups is due to random assignment—produces the simplest analysis strategy to show program effects

  7. Dissemination of Findings • Begin analyzing your data and publishing early—pay a writer to write up your results if you and your staff lack time • Early phase results might include: • Description of challenges faced by control group • Early evidence of climate change in college • As project matures (even at the end of second year phase) results might: • Compare control and treatment group progress in STEM courses, attitudes about STEM, other indicators relevant to your project? • Estimate effects on longer range goals of project and the possible mechanisms through which program works

  8. Where do you publish and disseminate your findings? • Publications of field specific education journals (the education arm of professional organizations, have journals and newsletters, e.g., Journal of Engineering Education) and education journals like Journal of Higher Education, Education and Evaluation Policy Analysis • Early findings and news briefs might go into professional newsletters • Work with your office of public affairs to prepare press releases • Consider an end-of-project book • Get ideas from other STEP projects

  9. Designing the study • What do you want to learn? How/why do you believe that the project succeed? • Who will you serve? What is your target population? • How do you recruit sample members and collect data?

  10. Designing the study • What exactly do you want to ask? • What will you do with all of those data once you have them?

  11. What do you want to learn? • Project will implement intervention you and NSF believe will work • What are the explicit objectives? • What are the interventions? • How/why will they work? (theory)

  12. What do you want to learn? • Example: Professor Yu ? Study groups Chemistry degree

  13. theory Knowledge Study groups Chemistry degree Enjoyment Self-confidence

  14. The target population • Potential chemistry and physics students • Latino and African American students 1. Recruitment: High school students 2. Tutoring: In selected classes 3. Study groups: In selected classes 4. Faculty mentor: in selected classes

  15. Who is in the control condition? • Recruitment: • Those not recruited? • In comparable non-participating schools? • Comparable non-participating classes? • Any way you cut it, must include kids in high school

  16. The target population • Latino and African American students 1. Recruitment: At selected schools 2. Tutoring: In selected classes 3. Study groups: In selected classes 4. Faculty mentor: in selected classes

  17. Peer tutoring • Target population: Students enrolled in gateway courses • Control group?

  18. Study groups • Target population: those enrolling in gateway courses in chemistry and physics

  19. Study groups • Control: • Entrants to same classes prior to project • Entrants to gateway courses in biology and computer science • Random assignment of sections to study groups or not

  20. Designing data collection • Mode of collection • Sampling procedures • Protocol for recruitment • Timing • The IRB

  21. Mode of collection • Survey • Interview (phone or in person) • Paper and pencil • Mail • In class • Online • Focus group

  22. Sampling • Identify target population • Locate/create sampling frame • Make sure you have contact data you need • Structure sample • Stratification • Matching • Blocking

  23. Sampling: The Yu study • Target population: Students enrolled in calculus 101, chemistry 101, biology 101 • Each course has 2-4 sections and each section 6-8 lab sections

  24. Sampling: The Yu study • Randomly assign lab sections to have study groups or not starting in spring of 2013 • Sample size of 600 • 200 students in fall of 2012 from all students enrolled in target courses • 100 treatment and 200 control students in spring of 2013

  25. Sampling: The Yu study • Threats to valid inference

  26. Recruiting the sample • Sample nonresponse undercuts validity • Shoot for at least 70% • Getting and maintaining sample is often the hardest part of the study • Without a good sample the rest of your evaluation efforts are wasted

  27. Recruiting the sample • Incentives for participation • There is actual research on this • Cash is king, other incentives (Amazon gift certificate, MP3 downloads, etc.) not very effective • Some evidence for effectiveness of lotteries, but weak

  28. Recruiting the sample • Initial contact • There is actual research on this • Letter on letterhead in stamped envelope if possible • In Yu’s case, possibly hand-delivered in lab

  29. Recruiting the sample • Non-response follow-up • Email • Snail mail • Phone

  30. Data collection timing • Consider last date you can collect data and work backwards • One week prior to exams • Snail mail incentive letter • 2 email follow ups • Snail mail follow up • One more email • After another week, phone calls

  31. Data collection timing • And of course the IRB…

  32. Instrument design • Goes back to what you want to know • Mechanisms by which intervention has effect • Variation in effects of intervention and what might account for that variation

  33. Instrument design • Do not make up your own items if you can avoid it • Consider extant instruments • Ask simple questions respondents can answer No ‘double barrel’ questions

  34. Instrument design • Survey is a ‘conversation with a purpose’ • Script the conversation so it flows • Use skip patterns to avoid asking irrelevant questions • Keep is as short as possible

  35. theory knowledge Study groups Chemistry degree Enjoyment Self-confidence

  36. Self-confidence: Identity How much do you agree or disagree with the following statements? You see yourself as a science person Others see you as a science person 1=Strongly agree 2=Agree 3=Disagree 4=Strongly disagree

  37. Self-confidence: Self-efficacy You are confident that you can do an excellent job on tests in this course You are certain you can understand the most difficult material presented in the textbook used in this course You are certain you can master the skills being taught in this course

  38. Analyzing Data • Depending on design • RCT: difference in means • Other designs: • Covariate adjustment • Difference in means for certain subgroups

  39. Analyzing Data • Our example: Does program impact self-confidence? • Mean change in self-confidence: • Control students in fall 2012 • Control students in spring 2013 • Treatment students in spring 2013 • Covariate adjustment

  40. List of journals for possible publication of STEP Type 1 results (list generated by audience) • Advances in Engineering Education - http://advances.asee.org/ • CBE Life Sciences Education - http://www.lifescied.org/ • Chemical Educator - http://chemeducator.org/ • Community College Journal of Research and Practice - http://www.tandf.co.uk/journals/authors/ucjcauth.asp • CUR Quarterly - http://www.cur.org/publications/quarterlies.html • Journal of Applications and Practices in Engineering Education - http://japee.net/ • Journal of Chemical Education - http://pubs.acs.org/journal/jceda8 • Journal of College Science Teaching - http://www.nsta.org/college/ • Journal of College Student Retention -http://www.cscsr.org/retention_journal.htm • Journal of Research in Science Teaching - http://www.narst.org/publications/jrst.cfm • Journal of STEM Education: Innovations and Researchhttp://ojs.jstem.org/index.php?journal=JSTEM&page=index • Journal of Women and Minorities in Science and Engineering - http://www.begellhouse.com/journals/00551c876cc2f027 • Physics Teacher - http://tpt.aapt.org/ • Physics Today - http://www.physicstoday.org/ • Portland International Conference on Management of Engineering and Technology (PICMET) - http://www.picmet.org/main/ • STEPcentral.net A Community Forum for NSF STEP Projects - http://stepcentral.net/ • Student Affairs in Higher Education - http://www.naspa.org/

More Related