Improving the Intelligence of Assessment Through Technology: An IES Perspective

1. Improving the Intelligence of Assessment Through Technology: An IES Perspective Martin E. Orland, Ph.D. Special Assistant Institute of Education Sciences United States Department of Education

2. Institute of Education Sciences • Established in late 2002 as replacement for OERI • Evidence-Based education as its cornerstone mission

4. New Features of IES • Evaluation Center (efficacy, impact) • Leadership (term, level) • Organizational flexibility • Greater independence (delegations, budget) • Regulatory freedom (FACA, regs) • Staffing (excepted service, internal reorg)

5. What is EBE? • The integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction

6. What is professional wisdom? • The judgment that individuals acquire through experience • Consensus views • Increased professional wisdom is reflected in numerous ways, including the effective identification and incorporation of local circumstances into instruction

7. What is empirical evidence? • Scientifically-based research • Empirical data to compare, evaluate, and monitor progress

8. Evidence-based Education

9. Why are both needed? • Without professional wisdom education cannot • adapt to local circumstances • operate intelligently in the many areas in which research evidence is absent or incomplete • Without empirical evidence education cannot • resolve competing approaches • generate cumulative knowledge • avoid fad, fancy, and personal bias

10. EBE -- Where are we?

11. Scientifically Based Research “…means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs” (No Child Left Behind Act of 2001)

12. National Research Council • Significant questions that can be investigated empirically • Links research to relevant theory • Uses methods to permit direct investigation of question • Provides coherent and explicit chain of reasoning • Replicate and generalize across studies and disclosure to promote scrutiny and critique

13. The Role of Assessment • Provides the empirical evidence on what students know, and what they have learned • Assumes valid and reliable sampling of such knowledge and learning

14. High Stakes for Assessment to “Get it Right” • Critical to successful implementation of SBR and attainment of the IES mission • Increasing relevance because of SBR, NCLB, and continued policy focus on accountability

15. Standard Critique of Educational Assessments • Mismatch between what can be measured efficiently and accurately and what is considered important • For practitioners, undertaken to meet “someone else’s” agenda (i.e., assessments are conducted apart from rather than a part of the regular business of teaching and learning) • Goals of assessment for accountability and instructional improvement viewed as being in fundamental conflict

16. Potential Consequences • Limited or inappropriate inferences for making both instructional decisions and policy determinations • The promise of SBR and data-driven decisionmaking not realized • Education is not transformed into an evidence-based field with heavy reliance on high quality empirical data

17. The Unique Promise of Technology to Assessment Reform • Input • To analyze and sort information on attributes of student learning and knowledge that should be most valued • Output • To facilitate the extraction of such information into a new generation of assessment tools that enhance student achievement (both weighing and feeding the calf)

18. The Challenge is Clear • Exploiting the Promise of Technology to Address Traditional Limitations of Educational Assessment • An integral tool for improving learning • Able to serve the needs of the educational practitioner and the policymaker • In Other Words, Making Assessments More “Intelligent”