Best Practices for Building Tests and Quizzes League for Innovation November 2004
7 question types • multiple choice • true/false • essay/short answer • matching • ordering • fill in the blank • multiple answers
Selected Response Questions • Selected response questions require students to select from a predetermined list of potential answers. • multiple choice • true false • Matching • fill-in-the blank questions
Created Response Questions • Ex. Essays measure the student’s ability to communicate effectively, not just their understanding of content. • Easier to write but harder and (more subjective) to grade.
Match Question Type to Level of Assessment Desired • Multiple choice and matching questions • offer the most flexibility in terms of content covered • thinking skills that can be assessed • True/false are usually limited to fact recall.
Match Instructional Objectives • Use lesson plans or teacher notes to accurately reflect content that was covered in class • Choose the most important objectives to assess and use these as the outline for your test
Cover Important Material • Facts, definitions, comprehension, analysis, applications • Trivial items result in trivial studying and learning.
Items Should be Independent • Do not “give away” answers via information in other questions • Independence maximizes breadth of coverage
Write Simply and Clearly • Measure knowledge of material and concepts, not vocabulary • Ambiguous questions create error, frustration, and compound biases related to language and disability
Clearly Specify What Type of Response is Sought • How long or short an answer is sought? • Should they show their work? • Do you want description? comparisons? application? evaluation? • Whose opinion do you want (book, lecture, their own)?
Good Tests Take Time to Write • Give yourself enough time to evaluate items after a day or two • Revise, edit, and ask others to read before administering the test
Good Tests Have High Degrees of Reliability and Validity • Reliability refers to the extent to which measurement is consistent • Validity refers to the extent to which a test measures what it should.
Construct Questions Carefully • The most important test-making process is selecting the wording of each question and answer. • Question Stem and Distractors
Construction Tips • Construct evaluation items with a single correct answer • Use plausible distracters • Arrange options in a logical sequence • Include a minimum of 3 and a maximum of 5 options • Alternate the order of the correct responses randomly among all options
Construction Tips • Refrain from using the choices “all of the above” or “none of the above” • Identify an initial difficulty level
Construction Tips • Increasing item difficulty can be achieved by paying attention to the distracters (options) • The more “homogeneous” the options and distracters, the higher the degree of difficulty
Higher Degree of Difficulty The first James Bond movie was released in what year? a) 1960 b) 1962 c) 1964 d) 1966
Mid-Level Difficulty The first James Bond movie was released in what year? a) 1960 b) 1962 c) 1964 d) 1970
Low-Level Difficulty The first James Bond movie was released in what year? a) 1958 b) 1962 c) 1970 d) 1975
Points of Concern for Different Formats • Writing True-False Items: • Choose words with precise, definite meanings • Avoid tricks and trivia • Avoid easy clues • Follow guidelines for multiple choice
Writing Multiple Choice Questions • Present the problem, including qualifying statement • There should be only one correct answer • Distracters should be plausible but clearly incorrect • Avoid negative wording (especially double negatives)
Writing Multiple Choice Questions • When item is controversial, indicate whose opinion is sought • Avoid irrelevant cues to correct answer (length, grammar) • Items should test one central idea or concept
Writing Multiple Choice Questions • Watch out for patterns in alternatives (overuse of "C" as correct answer) • Choose appropriate level of difficulty • Assure even coverage of material and types of knowledge
Writing Matching Items • Keep each matching set short (no more than 5 stimuli/responses) • Use longer items as stimuli, shorter as responses • Arrange responses in alphabetical or logical order
Writing Short Answer and Completion Items • Clearly indicate type of answer you want • Do not use more than two blanks per completion item • Make a key before scoring • Periodically re-score early tests to detect shifting criteria
Writing Essay Questions • Limit questions to vitally important material • Clearly define task, scope, and directions for a "good" answer • Allow time for thought • Use multiple medium-length essays rather than one long one
Writing Essay Questions • Use questions that have a limited number of good answers • Allow choice between alternatives (e.g., "answer 3 of the 4 questions")
Evaluating the Test • Once scored, spend some time reviewing the test and observing patterns that may be present. • Were there any questions that every student got wrong? • If so, can you deduce whether it was due to poor test item construction or to instruction?
Evaluating the Test • If poorly written, you may want to consider canceling it out and recalibrating the test. • If students didn’t understand the concept, perhaps the content needs to be retaught.
A Checklist for Teachers • Are your directions CLEAR and CONCISE? • Are your objectives clear? • Is there a logical connection between questions and answers? • Are you being specific in regard to the area you are covering? • Are your answers simple and factual?
Higher Order Thinking • What later events best affirm the ideas set forth in the Declaration of Independence? A) Emancipation Proclamation;19th Amendment B) Eminent Domain, Manifest Destiny C) Civil War
References • Orlich,Donald C, Harder,Robert J.,Callahan,Richard C, and Gibson,Harry W.(1998), Teaching Strategies: A guide to better instruction, 5th Ed, Houghton Mifflin, New York • Royse, David (2001) Teaching Tips for College and University Instructors: A Practical Guide, Allyn and Bacon, Boston
References • Dominowski, Roger L. (2002),Teaching Undergraduates, Lawrence Erlbaum Associates, New Jersey • Paul Eggen, Don Kauchak (2001),Educational Psychology, Windows on Classrooms, 5th Ed, Pub. Merrill- Prentice Hall, Upper Saddle River, New Jersey.
References • Bloom, B.S. (Ed.) (1956) Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green • Chase, C.I. (1999). Contemporary Assessment for Educators. New York: Addison Wesley Longman, Inc.
References • Dominowski, R.L. (2002). Teaching Undergraduates. Mahwah, NJ: Lawrence Erlbaum Associates. • Friedenberg, L. 1995. Psychological testing: Design, analysis, and use. Needham Heights: Allyn & Bacon.
References • Kay Burke (1999), The Mindful School: How to Assess authentic Learning Pub.SkyLight Professional Development , Arlington Heights, Illinois • Heinich, R., Molenda, M., Russell, J. D., and Smaldino, S. E. (1996). Instructional media and technologies for learning (5th ed.). (pp. 40-41). Englewood Cliffs: Prentice-Hall.
References • Jacobs, L. C. & Chase, C. I. (1992). Developing and using tests effectively: A guide for faculty. New York: Jossey-Bass. • Kopeikin, Hal S. (n. d.) UC Los Angeles lecturer, unpublished notes on writing effective questions. • http://web.wi.mit.edu/sive/pub/Lab%20Resources%20Documents/Thoughts%20on%20Exam%20Quests.html
References • http://www.rpi.edu/~verwyc/chap2tm.htm • http://www.glencoe.com/sec/teachingtoday/educationupclose.phtml/print/40 • http://taesig.8m.com/createv.html • http://www.utc.edu/Administration/WalkerTeachingResourceCenter/FacultyDevelopment/Assessment/test-questions.html • http://www.psych.ucsb.edu/~taap/writingtests.html(defunct)