Evaluating Technology’s Impact on Teaching and Learning

1. Evaluating Technology’s Impact on Teaching and Learning Educational Technology Integration

2. Contact Information • jsun@sun-associates.com • 978-453-3070 • www.sun-associates.com/eval/sample • This presentation is linked to that page

3. Where Do We Stand? • Hot topics • Your District • State • National • Your expectations for today

4. Workshop Goals • To overview a scheme for creating a formative evaluation of technology’s impact • To review related issues and developments visa vis our experience in a particular school district (Fayette County, KY) • To create an action plan for conducting a similar assessment in your own district

5. A Framework for Review

6. Why Evaluate? • To realize your investment in technology • What sort of “difference” has all of this technology made? • To continue your school or district technology planning effort • Evaluation is a key component of ongoing technology planning!

7. Evaluation Starts with Technology Goals • Evaluation should be rooted in a strong educational technology plan which... • Is more than an infrastructure plan • Focuses on technology’s impact on teachers and students • Has clear goals and objectives for what you want to see happen

8. Skeletal Plan Format

9. Your Goals? • Using the Plan Mapping Worksheet, map your plan in terms of vision, goals, and actions.

10. Goals Lead to Questions • What do you want to see happen? • These are your goals • Achieving these goals requires a process • Are you performing the process steps? • All of this can be measured through a formative evaluation

11. The Evaluation Process • Create meaningful evaluation questions • Design relevant indicators • Organize those indicators in a performance rubric which can be used to measure progress and achievement • Collect data to score the rubrics • Create a report which applies the evaluation in a formative manner

12. Take a break...

13. FCPS Example Questions • How have students been impacted by technology integration? • Are teachers using technology in ways that match district goals for technology and use the potential for instructional technology? • Have we adequately allocated district technology resources?

14. Processes of Technology Evaluation • What are you looking for? • Ways to make the implementation of technology better • Teaching and learning impacts — this is not about counting machines!

15. Developing Indicators • What is it that you want to measure? • Impact on teaching and learning • Change • Progress towards desired outcomes • You need indicators for progress to be measured

16. Indicators should reflect your school’s unique goals and aspirations • Arise from your plan • Rooted in your vision and goals • Indicators must be indicative of your unique environment...what constitutes success for you might not for someone else.

17. Indicators of Engaged Learning • Engaged learners are • Responsible for their own learning • Strategic problem-solvers • Energized by learning • Collaborative learners • These are all observable characteristics • These indicators can help inform your student impact rubric

18. Levels of Teacher Appropriation

19. Point is... There are things we can look for in terms of student and teacher behavior. • The goal in creating indicators is to link aspects of behavior to levels of achievement

20. What’s Realistic? • When developing questions and indicators, what makes sense visa vis the research on technology’s potential impacts? • www.sun-associates.com/eval/resources.html

21. What the Research Says • CBI generates measurable increases in mastery of core content • Kulik’s meta-analysis • Fletcher’s evaluation of military training • Pace, repetition, and standardization promote memorization of “facts” • Kulik’s work shows that CBI did virtually nothing in terms of changing student “attitude” toward subject matter

22. Productivity tools -- particularly tools for writing -- encourage students to produce more and pay more attention to the quality of their work. • This more free-form use of technology cuts directly to the heart of the connection between attitude toward learning and student achievement • Students will do more of what they like to do

23. Technology improves the learning environment • Motivates students • Makes learning more exciting • Makes learning more relevant to real-world tasks • None of these benefits come solely through technology!

24. It gets more gray when considering “cognitive uses” • Cooperative and collaborative tasks • Communication • Problem solving • Using technology as a key element of a rich, supportive, educational environment

25. Student Performance in Rich Educational Environments • Higher rates of attendance and graduation (ETS report) • Higher qualification rates for honors programs • Greater development of critical thinking and problem-solving skills • Generally higher scores on traditional assessments • Lower numbers of disciplinary incidents

26. These benefits have been observed/documented specifically in those classrooms where technology has been a key intervention...but never the only intervention!

27. What Other Interventions? • More teacher training • Usually in relation to using the technology tools as part of an overall reformed instructional environment • More individualized, student-centered, learning • Usually as a result of problem-based, cooperative learning environments where technology is the catalyst for change

28. So, What Changes? • The student outcomes from participation in a technology-enriched environment are indeed positive • Measures are usually in terms of new things that can be done versus increases on traditional assessments • Nevertheless, traditional assessments do show improvement...it just occurs over time and is part of a broader, system-wide, picture of change.

29. Try a Sample Indicator • Using the Developing Indicators worksheet, let’s develop a few sample indicators • Work in several groups • Select someone to report out • Try to develop some consensus on indicators which would work across districts

30. FCPS’s Indicators • Use of technology positively impacts and fosters the student’s motivation to engage in learning practices that lead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiring basic skills

31. The faculty and staff are proficient, knowledgeable, and current with available technology and translate that knowledge into relevant learning opportunities for students • Teachers create learning opportunities and physical environments that allow students to assume more independent roles in their own learning through their use of technology.

32. All teachers and learners throughout the district have sufficient access to technology-based productivity tools, on-line services, media-based instructional materials, primary sources of data, and adequate support for using these resources so as to enrich and extend their learning goals.

33. Rubrics www.fayette.k12.ky.us/central/edtech/suneval.asp

37. Rubric Tips • Even number of levels • Create from ideal to lesser • Yes, Yes But, No But, No • Be as descriptive as possible • Do not get excessively quantitative

38. Take a break...

39. FCPS Rubric Scores

40. Student Impact, 2 • Technology is being used daily as a basic tool within a range of traditional learning activities • Some evidence that technology is supporting new ways of thinking and learning • Most student technology use is very teacher-directed and occurs in settings outside of the classroom

41. Teacher Fluency • Teacher Fluency = 2.5 • Teachers are reasonably fluent, but need to do more to connect technology to the core curriculum • Use of Technology with Students = 2.25 • Teachers direct students to use technology, but need to use technology to empower students as independent, engaged, learners

42. Teacher Productivity = 4 • Teachers are very proficient in using technology as a tool for personal and professional productivity • Administrator Support = 2 • Administrators use technology, but need to advocate and support more integrated uses among their teaching staff

43. Infrastructure Allocation, 2.75 • The basic infrastructure -- hardware, network, software -- is complete and adequate • Individuals in Technology Support functions (schools and district) are doing an exemplary job and should receive much of the credit for the advances we have seen thus far • Schools need even more technology resource teacher support • Efforts must continue to map technology on to the core curriculum

44. Sum It All Up? • Fayette County is doing very well! • Based on our research nationwide, FCPS is clearly above average in most areas of its technology implementation • Teachers are using technology in ways that are consistent with their overall instructional approach • Directed instruction • Technology as a “skill” versus an integrated element of the learning process

45. Technology can do more, and teachers can do more • Professional development and increased levels of resource teacher support will make this happen • The full report contains a number of recommendations for professional development and policy designed to positively impact performance on the indicators

46. To Summarize... • Start with your plan • From your goals, develop indicators and a performance rubric • Accumulate data and/or develop your data collection tools • How will you collect the data necessary to document your stage of fulfillment?

47. Data Collection • Review Existing Data • Current technology plan • Curriculum • District/school improvement plans • www.sun-associates.com/eval/sample • Create a checklist for data collection

48. Collect New Data • Focus Groups/Interviews • Teachers • Parents • Students • Administrators • Other stakeholders • Classroom Observations

49. Surveys • Creating good surveys — what do you want to know • length • differentiation (teachers, staff, parents, community, etc..) • quantitative data • attitudinal data • timing/response rates (getting returns!) • www.sun-associates.com/eval/samples/samplesurv.html

50. Solicitation of teacher/parent/student stories • This is a way to gather truly qualitative data • What does the community say about the use and impact of technology?