Welcome to Test Item Construction and Evaluation

1. Welcome toTest Item Construction and Evaluation

2. Learning Objectives Upon completion of this lesson, participants will be able to: • Distinguish between assessment and evaluation. • Recognize examples and non-examples of appropriate multiple-choice test item construction. • Summarize the guidelines for constructing multiple-choice test items. • Define principles of evaluation. • Identify the criteria of a good evaluation. • Identify appropriate evaluation methods for specific outcomes. • Identify different types of course evaluation and evaluation data. • Describe methods for evaluation of a training module. • Prepare an end of course report. • Identify and plan self development needs as an instructor.

3. Definitions Educational Assessment: is the process of documenting, usually in measurable terms, knowledge, skills, attitudes, and beliefs. Educational Evaluation: is the evaluation process of characterizing and appraising some aspect/s of an educational process. Course Evaluation: is a questionnaire, which requires a written or selected response answer to a series of questions in order to evaluate the instruction of a given course. The term may also refer to the completed survey form or a summary of responses to questionnaires.

4. TEST ITEM CONSTRUCTION

5. Why Multiple-Choice? • MULTIPLE-CHOICE VERSATILITY Thorndike and Hagen (1969) have summarized the advantages of various essay and objective test formats. Positive attributes have been designated by plus signs in the table above, while distinct disadvantages have minus signs.

6. Overview • MULTIPLE-CHOICE ITEMS • Anatomy • Construction guidelines • Examples TEST

7. Anatomy of a Test Item • Stem + Alternatives = Test Item What is primarily responsible for the increase in the average length of life in the USA during the last fifty years? Compulsory health and physical education courses in public schools. The reduced death rate among infants and young children. * The safety movement. Which has greatly reduced the number of deaths from accidents. The substitution of machines for human labor. Stem Distractor Answer Alternatives Distractor Distractor

8. Advantages and Limitations • Advantages of multiple-choice test items • Good for many subject areas • Takes less time to answer • Less guessing • Easy to score

9. Advantages and Limitations • Limitations of multiple-choice items • Not good for displaying thought • Still subject to some guessing

10. Construction Guidelines • Constructed using a single written objective • Stem presents definite problem • Use simple and clear wording • Avoid clue words • Avoid grammatical giveaways • Alternatives of equal length • Highlight key words • Positively stated stems • Use as much wording as possible in stem • One clear best answer • Distractors are plausible and related

11. Examples Construct each item to assess a single written objective. Items that are not written with a specific objective in mind often end up measuring lower-level objectives exclusively or covering trivial material that is of little educational worth.

12. Examples POOR EXAMPLE • California: • Contains the tallest mountain in the United States. • Has an eagle on its state flag. • Is the second largest state in terms of area. • Was the location of the Gold Rush of 1849.* BETTER EXAMPLE • What is the main reason so many people moved to California in 1849? • California land was fertile, plentiful, and inexpensive. • Gold was discovered in central California.* • The east was preparing for a civil war. • They wanted to establish religious settlements.

13. Examples POOR EXAMPLE BETTER EXAMPLE The alternatives in the poor example above are rather wordy, and may require more than one reading before the student understands them clearly. In the better example, the alternatives have been streamlined to increase clarity without losing accuracy.

14. Examples POOR EXAMPLE • A word used to describe a noun is called an: • Adjective.* • Conjunction. • Pronoun. • Verb. BETTER EXAMPLE • A word used to describe a noun is called: • An adjective.* • A conjunction. • A pronoun. • A verb.

15. Examples POOR EXAMPLE BETTER EXAMPLE

16. Examples EXAMPLE

17. Examples Positively stated stems Negatively-worded items are those in which the student is instructed to identify the exception, the incorrect answer, or the least correct answer.

18. Examples POOR EXAMPLE BETTER EXAMPLE

19. Examples POOR EXAMPLE BETTER EXAMPLE

20. Examples POOR EXAMPLE • Which of the following artists is known for painting the ceiling of the Sistine Chapel? • Warhol. • Flintstone. • Michelangelo.* • Santa Claus. BETTER EXAMPLE • Which of the following artists is known for painting the ceiling of the Sistine Chapel? • Botticelli. • da Vinci. • Michelangelo.* • Raphael

21. Worksheet Write your objectives in this box. Keeps test item tied to a single objective. Write your test items in this box. Guidelines on the left will help you stay away from pitfalls that plague test writers.

22. Interim Summary • MULTIPLE-CHOICE ITEMS • Anatomy • Construction guidelines • Examples

23. Critical Elements • Data • Judgment • Educational objective • Predetermined standard

24. Types of Evaluation • Placement • Diagnostic • Formative • Summative

25. Criteria for Evaluation • Validity • Reliability • Objectivity • Comprehensiveness • Differentiation

26. Organization of the Test • Arrangement of test items • Arrangement of alternatives • Review test for accuracy • Change test periodically 

27. Interim Summary • Types of evaluation • Criteria of evaluation • Organization of the test

28. Effective Course Evaluation

29. Overview

30. Effective Course Evaluation • Lesson Outline: • What is evaluation? • The evaluation life cycle • Portions of course to be evaluated • Types of evaluation • Collecting evaluation data • Responding to significant feedback • End of course report • Instructor development

31. What is Evaluation? • Identifies and illuminates what is actually occurring in the area or activity that is the subject of evaluation • Produces sufficient information to enable the evaluator, using agreed upon criteria, to make valid and useful judgments about what is being evaluated

32. The Evaluation Life Cycle

33. Portions to Evaluate • Appropriateness of objectives for the learning group • Organization and sequencing of content • Effectiveness of the instructional methods • Quality of teaching and learning aids used • Validity of assessment methods used • Performance of students in the course • Overall course outcomes

34. Types of Evaluations • Process/Monitoring • Outcome • Impact • Needs Assessment • Structural • Cost-Benefit • Cost Effectiveness • Formative • Summative

35. Data Collection • Surveys • Interviews • Observation

36. Learner Feedback As learners are reaching the end of teaching or training, their experiences are particularly useful in terms of course evaluation.

37. Learner Performance • The following approach is recommended in evaluating test scores. • Accurately record trainee and class (average) scores. • Compare class results against the standards. • Check effectiveness of objectives. • Make notes for later analysis. • Identify students that bring down class average. • If the class is achieving or surpassing standard, continue to evaluate. 4th 5th 1st 2nd 3rd

38. Self Evaluation • In evaluating lessons, the following questions are a good starting point. • What worked very well in terms of student learning, and why was it so effective? • What am I least satisfied in the lesson and why? • How would I change it for next time?

39. Responding to Feedback • The following list shows some typical issues that may come up during evaluation. • Students find certain objectives too difficult to meet • Some objectives are outdated • Course materials are not sufficiently comprehensive • Certain parts of the course materials are confusing • Instructional methods may be uninteresting • Assessments are not fully testing all learning objectives • Assessment items are not correctly constructed

40. Preparing End of Course Report This feedback is critical to the development of the course. • Content of Report • Format of Report

41. Instructor Development • Address the following questions. • What are the specific competencies that need to be developed? • What options are available to develop these competencies? • How will I monitor and evaluate progress? • What indicators will I use to determine success?

42. Summary • What is evaluation? • The evaluation life cycle • Portions of course to be evaluated • Types of evaluation • Collecting evaluation data • Responding to significant feedback • End of course report • Instructor development

43. Next Session • Oct 9 – 10:30-12:00 • Methods of instruction and classroom delivery • Teaching techniques • Questioning techniques • Use of Video • Case Studies

45. Close Create a program you can be proud of!!