1 / 26

Research Design in SoTL: Commitment to Outcomes

This workshop clarifies the fundamental elements of four common quasi-experimental designs used in educational outcomes research, emphasizing the importance and challenges of effective research design. Participants will learn strategies to control for validity threats, identify ethical issues relevant to educational research, and understand the nuances of various assessment methods. By exploring both descriptive and inferential approaches, attendees will leave equipped with practical knowledge for designing impactful studies that enhance student learning and performance.

ting
Download Presentation

Research Design in SoTL: Commitment to Outcomes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014

  2. Research Design in SoTL

  3. Commitment to Outcomes Research

  4. Learning Objectives • Identify the basic elements of four common quasi-experimental designs used in outcomes research • Discuss strategies to “control” factors that might impact validity of conclusions drawn from designs • Be able to identify ethical issues in educational outcomes research and resources for IRB review

  5. Outcome Research is … • Important • Rewarding • Necessary

  6. And often… • Difficult • Time-consuming • Small effect sizes

  7. Type of Research Descriptive/Qualitative Inferential/Quantitative Question: What Works? hypothesis testing, confirmatory empirical, statistical, comparative Samples: large, representative Assessment: scores, rates deductive summative, precise, reliable • Question: What Is? • descriptive, generative • naturalistic, observational, constructivist • Samples: small, targeted • Assessment: interviews, observations • inductive • formative, “thick description”, expansive

  8. Formal Design • Focus of SoTL • Actions as educators • Student learning and performance • Question guides choice of design • Source Material: • Cook & Campbell (1979) – Quasi-Experimentation: Design & Analysis Issues for Field Settings • Trochim (2006) – Research Methods Knowledge Base • Online: http://www.socialresearchmethods.net/kb/ • NSF – User-friendly Handbook for Mixed Methods Designs • Online: http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153

  9. Optimal Design -- Descriptive • No single preferred design • Common features • Focus on individual and collective experiences of students • Context is described • Search for patterns • Developmental • Multiple assessment methods and assessment opportunities • Subgroup analysis

  10. Optimal Design -- Inferential

  11. Randomized “Clinical” Trial

  12. Real Life • Quasi-experimental designs • Non-random assignment to groups • Frequently used (and effective designs • Single group repeated measures design • Non-equivalent groups pre-test/post-test design • Cohort design • Single group crossover (ABAB) design

  13. Uninterpretable Designs • One-shot case study • Single-group pre-test/post-test design • Static group comparison X O O1 X O2 X O O

  14. Single Group Repeated Measures/Time Series • Add multiple observations to track learning/change • Goal is to study the trajectory of learning • Depth of knowledge • Understanding of concepts • Application • Integration/synthesis • Testing of ideas O1 O2 X O3 O4 O5 O6

  15. Non-equivalent Groups Pre-test/Post-test Design • Compare two classes • New vs. standard • Willing/interested colleagues • Commitment to method • Pre-test/Post-test essential • Targeted measurement O1 X O2 O1 O2

  16. Cohort Design • What if you are the only one who teaches this? • Or colleagues are not “willing” • Compare outcomes to yourself • But be very, very careful • Risk – nothing else can change • Assumes stable cohort O1 X O2

  17. Single Group Cross-over Design • Multiple units/cases • Replicate (ABAB) • Standard assessment strategy XA1O1O2O3 XB1O4O5O6 XA2O7O8O9XB2O10O11O12

  18. Less “Control” • Rule out competing explanations of findings • Argument/Logic • Measurement • Student demographics • Possible pre-test differences • Characteristics of “treatment” • Design • Statistical controls • Preventive action

  19. Sound Measurement • Existing standards • Independence of observations, where possible • Conceptual comparison • Multiple dimensions/skills

  20. Your Turn… • Choose design that best “fits” your case • Type of class • Possibility of comparison • Assessment strategy

  21. IRB Review • Educational research • Human participants • Type of research • Non-research/exempt/expedited • Primary Issues • Coercion • Informed consent

  22. Everything IRB • www.irb.emory.edu • Social-behavioral studies • Multi-disciplinary team • Rebecca Rousselle • Carol Corkran

  23. Non-Research • New category • Systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge • PI can make this determination • If it is just for your own evaluation, it is not research

  24. Exempt vs. Expedited • Most surveys, interviews, or observations of public behavior are presumed exempt if the participant cannot be identified (no links from the person to the data) or the responses/data can not harm the participant. • IRB determines this

  25. Coercion • Major issue • Students must be free not to participate

  26. Informed Consent • Description of the purpose of the study • Who is being studied • What will be required • Risks/benefits • Compensation • Legal review • Contact information • Summary of findings

More Related