Evaluating the Effectiveness of Educational Technology

1. Evaluating the Effectiveness of Educational Technology Distance Teaching and Learning Academy, July 2000 Dakota State University Mark Hawkes

2. Learning Objectives • Understand the nature of program evaluation and its role in determining the value of educational technologies • Observe some options for conducting evaluations • Introduce the issues and options of student assessment with educational technology

3. Evaluation: Definition “Determining the worth or merit of something.” Scriven, 1967

4. Evaluation Uses • To assist decision-makers responsible for making policy • To satisfy external funding agencies • To identify learning needs and set priorities among needs • Identify appropriate program approaches, organizations, staff assignments, materials and equipment . . . • Determine requests for resources

5. Dimensions to Evaluation • Formal vs. Informal • Formative vs. Summative • Internal vs. External

6. Models of Evaluation • Objectives-oriented • Consumer-oriented • Expertise-oriented • Participant-oriented • Adversary oriented

7. Systems: Technology infrastructure Equity in use Operability/compatibility Interactivity Applications Learning outcomes Learner: Cognitive process Teacher development Motivation Learning outcomes Domains of Ed. Tech. Evaluation

8. Which of the following methods does your school use to evaluate the impact of technology on student achievement? (Superintendents/principals)  43% student performance on achievement tests  61% teacher feedback ·      12% school board or community support ·     11% third-party independent evaluation ·     9% vendor provided tests  7% don’t know American Association of School Administrators Lightspan Partnership, Inc.

9. What evidence have you seen that technology is improving achievement for low-performing students? • 30% increased test scores • 42% increased mastery of content • 60% increased student motivation • 28% decreased behavioral problems American Association of School Administrators Lightspan Partnership, Inc.

10. Which of the following ways that schools invest in technology receives the greatest share of your budget? •  84% hardware/computers •  44% instructional software •  26% teacher training in the use of computers and software •  25% wiring/infrastructure •  7% Internet access American Association of School Administrators Lightspan Partnership, Inc.

11. Which of the following technology purchases do you expect to have the greatest impact on student achievement?  31% instructional software  28% teacher training in the use of computers and software ·      20% hardware/computers ·      14% Internet access  5% Wiring/infrastructure American Association of School Administrators Lightspan Partnership, Inc.

12. Which of the following would be the most effective way to increase the impact of instructional software on student achievement?  42% provide additional instruction to teachers on the use of software ·        29% ensure software ties to curriculum objectives ·        16% provide adequate time for students to use software  12% provide more teacher choice in selecting software American Association of School Administrators Lightspan Partnership, Inc.

13. Basic Organizing Questions (BOQ’s) • What questions should I ask? • Episcopal HS, Alexandra, VA • Tech High TCG—Pulaski, KY • What information do I need? • Waukegan, IL—TCG • What’s the best way to get the information? • Blackfoot, ID Technology Evaluation Plan • How do I communicate the results?

14. Evaluation in your district . . . You are assigned to evaluate the technology-focused professional development activities in your district. Identify three key questions to guide your evaluation and the sources and methods you’ll use to answer those questions.

15. Demand-Driven Teacher DevelopmentTeacher Professional Development, Technology and Evaluation. A paper presented at the American Evaluation Association Annual Conference, Orlando, FL. November, 1999. • Tied to real-live teaching concerns • Encourages reflective understanding • Encourages peer interactivity • Models the inquiry process • Solution intentioned • Mediates new forms of teacher/student engagement

16. Evaluation Criteria (cont.) Does the Teacher Development . . . • Model authentic, inquiry-based engaged learning? • Honor the K-12 teacher knowledge-base? • Encourage teachers to integrate and share documentation of their practice? • Respond to contextual issues of technology use? • Show how other teachers address similar problems and situations?

17. Evaluation Criteria Does the Teacher Development . . . • Provide broad access to communication tools? • Encourage teacher understanding of the research process? • Involve teacher in collaborative, knowledge building communities? • Offer just-in-time support? • Show the boundary’s of technology’s capacity? • Address personal assumptions about teaching, learning and schooling?

18. Evaluating Learning • Most research on technology and student achievement has used traditional standardized assessments to measure changes in student performance • Newer standardized tests may be appropriate if they de-emphasize students' knowledge of isolated facts and pay attention to how well students think

19. Good Assessment • Measures the intended outcomes • Athentic—focused on problem-solving • Motivates students to learn • Incorporates improvement for toward self-learning

20. Alternative Assessment • Includes: essays, performance assessment, oral presentations, demonstrations, exhibitions, and portfolios. • Product oriented • Reinforces content learning

21. Why use technology to assess? • Assessment tasks using technology replicate many tasks students will perform as adults • Forces students to use knowledge, not just memorize it • Creates a classroom based more on formative than summative outcomes • Allows time for collaborative projects • Promotes self and peer assessment • Improves capacity to archive assessments • Improves the nature and amount of teacher feedback

22. Disadvantages to Assessment with Technology • Takes time to teach the students the skills to use technology • Generally takes longer • Increases teacher preparation time • May present grade equality questions • Technology is not always available

23. Examples . . . • Bellingham Public Schools • 5th Grade Assessment • Performance Task: Inventions • Performance Task: Student Directions • Directions for Test Administrator • Final Report Form • 11th Grade Assessment • Task Directions: Accident Prevention • Final Report Form

24. Example: Image Editing Software Grade Level: 6th to 7th grade Higher Level Thinking Skills: Synthesis and Evaluation Illustration: Create a collage of pictures that portrays the causes of the U.S./Soviet Cold War

25. Example: Access/ITV Grade Level: 9th grade Higher Level Thinking Skills: Application, Synthesis, Evaluation Illustration: Design a database to analyze the results of a student poll on crime in your community. Exchange that data with students from a neighboring community.

26. Example: ITV Grade Level: 10th grade Higher Level Thinking Skills: Application, Analysis, Evaluation Illustration: Create a video tutorial that teaches other students good study skills

27. Example: ITV Grade Level: 12th grade Higher Level Thinking Skills: Synthesis, Analysis, Evaluation Illustration: Create a commercial video solution for attracting industrial partners into development collaborations with your community

28. Critical Issues in Evaluating the Effectiveness of Technology • The effectiveness of technology is embedded in the effectiveness of other school improvement efforts. • Current practices for evaluating the impact of technology are not broad enough. • Standardized test scores offer limited formative information with which to drive the development of a school's technology program. • Schools must document and report their evaluation findings in ways that satisfy diverse stakeholders' need to know.

29. Critical Issues . . . • School and researchers need a common language and standards of practice for measuring how schools achieve that effectiveness. • The role of teachers is crucial in evaluating the effectiveness of technology in schools, but the burden of proof is not solely theirs. • Existing policies need to be "transformed" to match the new needs of schools using technology.

30. Issues to consider . . . • To what extent should evaluation attempt to shape development of technology programs? • How are these criteria/questions tied to larger issues of school reform, if at all? • What evaluation methods best respond to these types of criteria?

31. Resources on Technology Evaluation • Critical Issues in Evaluating the Effectiveness of Technology, McNabb, Hawkes & Rouk (1999). US DOE Office of Educational Technology. • An Educator's Guide to Evaluating The Use of Technology in Schools and Classrooms, Quiñones & Kirshstein (1998). US DOE Office of Educational Technology. • Criteria for Evaluating School-Based Distance Education Programs, Hawkes (1997). Journal of the National Association of Secondary School Principals. • Evaluating School-Based Distance Education Programs: Some Thoughts About Methods, Hawkes (1997). Journal of the National Association of Secondary School Principals.

32. Presentation Web Address http://www.homepages.dsu.edu/ hawkesm/Presentations.htm