Overview. Why Evaluate?Types of Assessment Theoretical FrameworkEvaluation Research Questions Methods
1. Evaluation and Assessment Strategies for Web-Based Education Paula Frew, MA, MPH
Associate Director of Programs
Behavioral Sciences and Health Education
Atlanta, Georgia, USA
2. Overview Why Evaluate?
Types of Assessment
Evaluation Research Questions
Methods & Data Collection Strategies
Frew (2001) Formative Evaluation Study of Web-Based Course Initiatives
3. Introductions About us
About the workshop leader
Ice-breaking exercise - getting to know each other
5. Why evaluate? Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object (program, technology, activity, etc.) - William Trochim,Cornell University (http://trochim.human.cornell.edu/kb/intreval.htm)
Learner evaluation: Assess worth or quality of learning and instruction in web-based environment (Donald E. Hanna, et al., 2000)
6. Why Evaluate? To improve a program/course
Assess outcomes and efficiency - Provide “useful feedback” to administrators, faculty, sponsors, other relevant constituencies.
Find out how a program operates/Understand why a program works (or does not) - May aid in decision-making or policy formulation processes through provision of empirically-driven feedback.
7. Why evaluate? Other reasons:
To generate knowledge - assess the academic/applied value which includes topics such as:
Distinguishing types of interventions
Learning about measurements
8. Why evaluate learners? To identify content areas that are unclear and confusing
Recognize content areas that need revision
Gather evidence to support revisions
Assess effectiveness of your course
(Hanna, et al., 2000)
9. Reflective Exercise: Your reasons to evaluate Workshop exercise - what are your reasons to conduct an evaluation?
Detail specifics related to own experience
Who are your stakeholders?
What is the political dynamic at work?
What is your relationship to the stakeholders and others involved in process?
Conclude exercise: Share with others at workshop
10. Types of Assessment Improvement-oriented evaluations:
Formative evaluation - strengthen or improve the object being evaluated*
Scriven (1967) - form or develop programs
Patton (1994) - “developmental evaluation”
Focus on processes - what is going on
Quality-enhancement and continuous improvements
Local adaptation (for different cultures)
11. Types of Assessment Knowledge-oriented evaluations:
12. Types of Assessment Judgement-oriented evaluations:
Summative evaluation - examine the effects or outcomes of some object*
Robson (2000) - an “end of term report” - what goals have been achieved?
Were needs met, target audiences reached, was program implemented as planned?
Audit: accountability and quality control
13. Program/Technology/Courses: Formative Evaluation Distinctions Needs assessment - who needs the program, how great is the need, what might work to meet the need?
Structured conceptualization - helps define the program/technology, the target audience, and possible outcomes
Implementation evaluation - monitors the fidelity of the program or technology delivery
Process evaluation - investigates the process of delivering the program or technology, including alternative delivery procedures (Trochim, 1999)
14. Formative Evaluation: Learning & Instructional Focus Helps to identify the knowledge and skills learners have gained in the course to date
Allows you to determine whether or not to introduce new content
Gives you feedback on the learners’ learning processes
Signals whether learners need additional practice or work in certain areas
Refocuses the learners’ attention
15. Program/Technology/Courses: Summative Evaluation Distinctions Outcomes evaluation - investigate whether the program or technology caused demonstrable effects on specifically defined target outcomes
Impact evaluation - assesses the overall or net effects of a program or technology as a whole
Cost-effectiveness/Cost-benefit analysis - address questions of efficiency by standardizing outcomes in terms of their dollar costs and values
Meta-analysis - integrates the outcomes estimates from multiple studies to arrive at an overall summary judgement on an evaluation question (Trochim, 1999)
16. Summative Evaluation: Learning & Instruction Focus Measures what learners have learned
Finalizes decisions about grades
Reviews new knowledge and skills the learners have gained in taking the course.
17. Reflective Exercise: What type of evaluation do you propose? Given the perceived need of the stakeholders, what type of evaluation would you pursue? Why?
What challenges do you perceive in conducting this type of evaluation?
What would be gained in doing this type of evaluation?
Exercise conclusion: share with others
18. Theoretical Framework 3 Approaches (Patton, 1994)
Deductive Approach - scholarly theory guides inquiry
Inductive Approach - generate theory from fieldwork
User-focused - working with others in evaluation context to extract and specify their implicit theory of action
19. Theoretical Framework Deductive Approach Examples:
Diffusion of Innovations Theory (Rogers, 1995) - e.g., examine the adoption of educational innovations in academic settings
Adult Learning Theory (Knowles, 1980) - e.g., how learners acquire their skills, knowledge, understandings, attitudes, values and interests
20. Reflective Exercise: What Theoretical Approach will you take? What is your theoretical orientation, if any, in conducting the evaluation?
Reasons for this decision
Potential blindspots in theoretical orientation
Exercise conclusion: share with others
21. Evaluation Research Questions Formative Research Examples:
How should the program or technology be delivered to address the problem?
How well is the program or technology delivered?
What is the definition and scope of the problem/issue, or what is the question?
Where is the problem and how big or serious is it? (Trochim, 1999)
22. Evaluation Research Questions Learning & Instruction: Formative Evaluation Examples:
How well is the instruction likely to work?
What obstacles are learners encountering, and how can they be overcome?
Are the selected instructional methods, media, and materials effective in helping learners learn?
What improvements could be made in the instruction for future use?
(Hanna, et al., 2000)
23. Evaluation Research Questions Summative Evaluation Research:
What was the effectiveness of the program or technology?
What is the net impact of the program? (Trochim, 1999)
24. Evaluation Research Questions Learning & Instruction: Summative Evaluation Examples:
What did the learners learn?
Did the learners find the instruction interesting, valuable, and meaningful?
What instructional methods contributed to learner motivation?
(Hanna, et al., 2000)
25. Reflective Exercise: Your research questions Write up to 5 questions that will guide your evaluation work - what do you hope to learn from these questions?
Once answered, what impact do you see this information having on your intended audience (the stakeholders, institution, etc.)?
Conclusion: share with others
26. Program/Course/Technology Evaluation: Methods & Data Collection Strategies Most used:
Formal, open-ended interviews
Formal, questionnaire-based interviews
Document review & analysis
27. Learner Evaluation: Data Collection Strategies Most used:
Pre-test/post-test quizzes of knowledge and skills
Performance evaluations/learner self-assessment
Learner participation figures
28. Reflective Exercise: Your methods/data collection approach What methods would you employ in your evaluation study? Why?
How much time and resources do you think are necessary to conduct your evaluation?
How would you propose to address bias in your data collection approach?
Conclusion: share ideas about your methods with others
29. Quick Tips on Questionnaire/Survey Design A popular method in gathering data for evaluation studies
Tips on writing survey questions (handout distributed to participants)
Statistical implications - how questionnaire data is turned into statistics/how to avoid bias in writing questions
30. Reflective exercise: Write a brief questionnaire for your evaluation study Write 5 good questionnaire items for your evaluation survey based upon your research questions
Who is your intended audience for this instrument (students?, faculty?, administrators?) - why does this matter?
Conclusion: share questions with others/why is it difficult to write good questions?
31. If time, exploring other methods Role play exercise: conducting a focus group
Role play exercise: how to interview participants
Basic descriptive statistics: how to analyze user statistics for web-based courses and programs
32. Frew (2001): Formative Evaluation Study of Web-Based Course Initiatives Why Evaluate: Are we progressing toward meeting Teaching Strategic Plan objectives?
What Type of Evaluation: Formative - focus on improving course website to meet specific TSP goals Research Questions: example - to what extent do the participants agree that the website enhanced the course curriculum?
Theoretical Orientation: DOI and ALT
33. Frew (2001): Formative Evaluation Study of Web-Based Course Initiatives Study Population:
Faculty and Medical Students (MEDI 605 course - Fall 2000)
34. Frew (2001): Formative Evaluation Study of Web-Based Course Initiatives Results (an example):
Curriculum Enhancement Question:
72.2% faculty - increased enthusiasm for instructional resource
94% students - website improved quality of course
Recommendations for Future Educational Development (examples):
Greater faculty involvement in diffusion process
Offer pre-course training
Improve access to computing resources
35. Additional Resources: Hanna, D.E., Glowacki-Dudka, M., & Conceição-Runlee, S. (2000). 147 Practical Tips for Teaching Online Groups: Essentials of Web-Based Education. Atwood Publishing: Madison, WI.
Knowles, M.S. (1980). The Modern Practice of Adult Education: From Pedagogy to Andragogy. Cambridge/Prentice Hall: Englewood Cliffs, NJ.
Patton, M. (1994). Utilization-focused Evaluation: 3rd Edition. SAGE Publications: London.
Robson, C. (2000). Small-Scale Evaluation. SAGE Publications: London.
Rogers, E.M. (1995). Diffusion of Innovations (4th ed.). The Free Press: New York.
Trochim, W. (1999). The Research Methods Knowledge Base, 1st Edition. Atomic Dog Publishing: Cincinnati, OH.
36. Discussion: Your Evaluation Experiences: Lessons Learned Questions & Answers