1 / 28

IES Scientific Standards For Fast Response Projects

IES Scientific Standards For Fast Response Projects. Presentation to the Regional Educational Laboratory Directors Meeting February 8, 2008. Fast Response Projects Provide. Scientifically based information on… Issues important to regional decision makers.

koto
Download Presentation

IES Scientific Standards For Fast Response Projects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. IES Scientific Standards For Fast Response Projects Presentation to the Regional Educational Laboratory Directors Meeting February 8, 2008

  2. Fast Response Projects Provide • Scientifically based information on… • Issues important to regional decision makers

  3. NCLB Act Established IES Scientific Standards • Use rigorous, systematic, objective methods • Ensure findings are appropriate to and supported by methods

  4. NCLB Act Further Specifies Scientific Research Involves • Systematic empirical methods • Research designs appropriate to question • Reliable measurement, appropriate analyses • Causal claims only from designs supporting them • Replicability • Peer review

  5. Guiding Principles of Fast Response Projects 1. Attend to regional needs • Know the audience and purpose 2. Select clear, focused research questions 3. Adhere to IES research standards • Appropriate data sources • Valid, reliable analysis or synthesis • Appropriate, reasonable conclusions

  6. An Issues and Answers Report… • Describes a problem, frames a research question • Briefly describes data and analysis methods • Presents findings (and limitations) • Appendices • Allow researchers to assess evidence • Provide supplementary data

  7. ATS Peer Review Asks Three Questions 1. Is research question clear, linked to policy issues and knowledge base? 2. Are methods and data appropriate, clearly described? 3. Are findings clearly presented, appropriate?

  8. IES Standards:1. Framework and Background

  9. Building a Justification for the Research • Report addresses an information need of the region • Interest beyond researchers • Study presents new information • Or makes new use of existing information

  10. Use of Prior Literature to Motivate Report Should Be • Balanced • Conflicting evidence • Key publications • Scientific • Discuss strengths and limitations, quality of evidence • Use primary sources

  11. When Framing Research Questions • State all questions clearly and in one place • Make them specific • Link questions to the policy issue and regional information need • Describe how answers are useful for the region

  12. Use Key Terms for a Broad Readership • Define terms early • Use definitions broadly accepted by researchers . . . • . . . that are clear and understandable for non-researchers • Use terms consistently

  13. IES Standards:2. Data Sources and Methods

  14. Sources, Data Collection Methods, Sampling • Explain where/who data came from, how you got it, appropriateness for research questions • How consistency achieved across collectors • Describe how you selected sample • Address representativeness; explain outreach • Basics in main body, details in appendix

  15. Analysis Methods • Make clear why things were done, not done • Explain how analysis methods are appropriate for research questions • Make clear how findings were derived

  16. Specific Analytic Approaches • Quasi-experimental, regression discontinuity studies • Must follow WWC guidelines (evidence screens, standards) • Other quantitative analyses • Cannot address causality • Provide details on multivariate models

  17. Specific Analytic Approaches (continued) • Literature reviews • Addressing causality - must follow WWC guidelines (evidence screens, standards) • Not addressing causality - search process, inclusion rules must be clear, systematic

  18. Specific Analytic Approaches (continued) • Qualitative analysis (e.g., documents, interview or focus group data) • Explain coding process, inter-rater reliability • Specify unit of analysis • Make clear extent to which participants had chance to (did) respond • Alignment study (curricula, standards, assessments) • Use dimensions that are objective, appropriate to questions

  19. Strengths and Limitations • Germane to findings • Address internal validity, generalizability • Don't ignore limits when presenting findings

  20. OMB Issues and Confidentiality • Similar data from no more than 9 respondents • No individually identifiable data on students, families, teachers, other staff • No identifiable data on schools, districts that are not publicly available

  21. IES Standards:3. Presentation of Findings

  22. Answer Research Questions • Align findings with the study’s research questions • Address all the research questions adequately • Make sure readers can find answers easily

  23. Support the Findings • Make sure findings clearly supported by the data and analysis • Describe the findings accurately and objectively • Without “un-scientific” adjectives

  24. Present Findings Clearly • Organize presentation of findings in a clear, logical fashion • Follow framework supported by literature presented earlier • Make sure reader knows how each finding helps answer a research question

  25. Discuss the Implications of the Findings • Show how the findings confirm or disconfirm prior research • Make sure suggestions for further research are feasible, and follow from the findings • Let decision makers decide

  26. The Review Process

  27. Report Reviews: First Draft ATS Fast Response Lead Liaison between reviewers and IES Reviewer A (ATS Internal)Answers review questions Reviewer B (External Expert) Answers review questions Summary author (ATS Internal) Third reviewer; synthesizes main issues in a summary

  28. Report Reviews: Revised Drafts ATS Fast Response Lead Liaison between reviewers and IES Primary Reviewer (1 of 3 first round reviewers) Writes review; areas of assessment primarily driven by first round review Secondary Reviewer (1 of 3 first round reviewers) Adds to, clarifies, questions review of Primary Reviewer

More Related