Slide1 l.jpg
Advertisement
This presentation is the property of its rightful owner.
1 / 39

Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional Development Fall 2005 PowerPoint PPT Presentation

Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional Development Fall 2005 Dr. Kristi Roberson-Scott Guiding Principles for Evaluation Evaluation should relate directly to instructional objectives

Related searches for Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional Development Fall 2005

Download Presentation

Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional Development Fall 2005

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional DevelopmentFall 2005

Dr. Kristi Roberson-Scott


Guiding principles for evaluation l.jpg

Guiding Principles for Evaluation

  • Evaluation should relate directly to instructional objectives

  • Each evaluation activity should be designed to promote student growth

    • The actual activity should be useful practice in itself

    • Feedback should be useable by the student

  • Multiple evaluation strategies should be provided to master achievement of X objective/competency

  • Student should clearly understand the methods of evaluation for X test or activity


Questions to ask yourself in designing a test l.jpg

Questions to Ask yourself in Designing a Test

  • What objectives will (should) I be testing?

  • What types of items will be included in the test?

  • How long will the test be in terms of time and number of items?

  • How much will each objective be worth in terms of weighting and number of items?


Tests as diagnostic tools l.jpg

Tests as Diagnostic Tools

  • Students demonstrate learning

  • Instructor effectiveness – modify teaching strategies or activities

  • Assignment of letter grades


Different types of tests learning l.jpg

Different Types of Tests & Learning

  • Paper & Pencil/WebCT Testing

    • Limited Choice Questions (MC, T/F, Matching)

    • Open-Ended Questions (Short Answer, Essay)

  • Performance Testing

    • Acquisition of skills that can be demonstrated through action (e.g., music, nursing, etc.)


Planning a test l.jpg

Planning a Test

  • First step: Outline learning objectives or major concepts to be covered by the test

    • Test should be representative of objectives and material covered

    • Major student complaint: Tests don’t fairly cover the material that was supposed to be canvassed on the test.


Planning a test7 l.jpg

Planning a Test

  • Second Step: Create a test blueprint

  • Third Step: Create questions based on blueprint

    • Match the question type with the appropriate level of learning

  • Fourth Step: For each check on the blueprint, jot down (might use 3x5 cards) 3-4 alternative question on ideas and item types which will get at the same objective

  • Fifth Step: Organize questions and/or ideas by item types


Planning a test8 l.jpg

Planning a Test

  • Sixth Step: Eliminate similar questions

  • Seventh Step: Walk away from this for a couple of days

  • Eighth Step: Reread all of the items – try doing this from the standpoint of a student


Planning a test9 l.jpg

Planning a Test

  • Ninth Step: Organize questions logically

  • Tenth Step: Time yourself actually taking the test and then multiply that by about 4 depending on the level of students

  • Eleventh Step: Analyze the results (item analyses)


Translating course objectives competencies into test items l.jpg

Translating Course Objectives/Competencies into Test Items

  • Syllabus

    • Specification table- what was taught/weight areas to be tested

  • Creating a Test Blueprint (see handout)

    • Blueprint- this is the test plan, i.e., which questions test what concept

    • Plotting the objectives/competencies against some hierarchy representing levels of cognitive difficulty or depth of processing


Thinking skills l.jpg

Thinking Skills

  • What level of learning corresponds to the course content

  • Bloom’s Taxonomy of Educational Objectives

    • Knowledge (see handout)

    • Comprehension

    • Application

    • Analysis

    • Synthesis

    • Evaluation


Practical considerations l.jpg

Practical Considerations

  • Representative sample of the course content not random– purposeful based on blueprint

  • Representative sample of skill or cognitive levels across content

  • Analyze results by level AND content area


Question arrangement on a test l.jpg

Question Arrangement on a Test

  • Group by question type

    • Common instructions will save reading time

  • Limit the number of times students have to change frame of reference

  • Patterns on test must be logical

    • Arrange from a content standpoint

    • Keep similar concepts together

  • Group by difficulty (easy to hard)


Selecting the right type of evaluation l.jpg

Selecting the Right Type of evaluation

  • How do you know what type of question to use and when?

  • It depends on the skill you are testing.

  • Evaluation should always match as closely as possible the actual activity you’re teaching.

    • Examples: Teaching Speech, should evaluate an oral speech

    • If testing ability to write in Spanish, better give an essay.

    • Testing reading –MC, TF

    • Wouldn’t use MC to test creative writing


Question types verses cognitive levels of learning l.jpg

Question Types verses Cognitive Levels of Learning


Constructing the test l.jpg

Constructing the Test

  • Types of Test Questions:

    • Multiple-Choice Items

    • True-False Items

    • Matching Items

    • Fill-In, Completion or Short-Answer Items

    • Essay Questions


Multiple choice items l.jpg

Multiple Choice Items

  • Advantages:

    • Extremely versatile-can measure the higher level mental processes (application, analysis, synthesis and evaluation)

    • A compromise between a short answer/essay and T/F item

    • Can cover a wide range of content can be sampled by one test

  • Disadvantages

    • Difficult to construct plausible alternative responses


Types of multiple choice items l.jpg

Types of Multiple Choice Items

  • Four Basic Types

    • Question Type

    • Incomplete Statement Type

    • Right Answer Type

    • Best Answer Type

  • Which Type is Best?

    • Question Type vs. Incomplete Statement

    • Right Answer vs. Best Answer Type


Multiple choice items19 l.jpg

Multiple Choice Items

  • Writing the stem first:

    • Be sure the stem asks a clear question

    • Stems phrased as questions are usually easier to write

    • Stems should not contain a lot of irrelevant info.

    • Appropriate reading level/terms

    • Be sure the stem is grammatically correct

    • Avoid negatively stated stems


Multiple choice items20 l.jpg

Multiple Choice Items

  • Writing the correct response

    • Use same terms/reading level

    • Avoid too many qualifiers

    • Assign a random position in the answer sequence

  • Read the stem and correct response together

  • Generate the distractors/alternative responses


Multiple choice items21 l.jpg

Multiple Choice Items

  • Other Tips for Constructing MC Items:

    • Items should have 3-4 alternatives.

    • Stem should present a single, clearly formulated problem

    • Simple, understandable, exclude extraneous words from both stem and alternatives

    • Include in the stem any word that are repeated in each response

    • Avoid all of the above (can answer based on partial information)

    • Avoid none of the above


Multiple choice items22 l.jpg

Multiple Choice Items

  • Alternative responses/distractors should be plausible and as homogeneous as possible

  • Response alternatives should not overlap

    • Two synonymous terms (arithmetic average/mean)

  • Avoid double negatives

    • None of the following are part of the brain except which one?

  • Emphasize negative wording

  • Each item should be independent of other items in the test

    • Information in the stem of one item should NOT help answer another item.


True false test items l.jpg

True-False Test Items

  • Best suited for testing 3 kinds of info.:

    • Knowledge level learning

    • Understanding of misconceptions

    • When there are two logical responses

  • Advantages:

    • Sample a large amount of learning per unit of student testing time

  • Disadvantages:

    • Tends to be very easy

    • 50-50 chance of guessing

    • Tends to be low in reliability


  • Tips for constructing true false items l.jpg

    Tips for Constructing True/False Items

    • Tips for constructing True-False Items

      • Avoid double negatives

      • Avoid long or complex sentences

      • Specific determiners (always, never, only, etc.) should be used with caution

      • Include only one central idea in each statement

      • Avoid emphasizing the trivial

      • Exact quantitative (two, three, four) language is better than qualitative (some, few, many)

      • Avoid a pattern of answers


    Objective test item analyses l.jpg

    Objective Test Item Analyses

    • Evaluating the Effectiveness of Items..

      • Why?

        • Scientific way to improve the quality of tests and test items

        • Identify poorly written items which mislead students

        • Identify areas (competencies) of difficulty

      • Item analyses provided info. on:

        • Item difficulty

        • Item discrimination

        • Effectiveness of alternatives in MC Tests


    Short answer items l.jpg

    Short-Answer Items

    • Two Types: (Question and Incomplete Statement)

    • Advantages:

      • Easy to construct

      • Excellent format for measuring who, what, when, and where info.

      • Guessing in minimized

      • Student must know the material- rather than simply recognize the answer

    • Disadvantages:

      • Grading can be time consuming

      • More than one answer can be correct


    Short answer items27 l.jpg

    Short Answer Items

    • Tips for Constructing Short Answer Items

      • Better to supply the term and require a definition

      • For numerical answers, indicate the degree of precision expected and the units in which they are to be expressed.

      • Use direct questions rather than incomplete statements

      • Try to phrase items so that there is only one possible correct response

      • When incomplete statements are used, do not use more than one blank within an item.


    Essay questions l.jpg

    Essay Questions

    Types of Essay Questions

    • Extended Response Question

      • Great deal of latitude on how to respond to a question.

      • Example: Discuss essay and multiple-choice type tests.

    • Restricted Response Question

      • More specific, easier to score, improved reliability and validity

      • Example: Compare and contrast the relative advantages of disadvantages of essay and multiple choice tests with respect to: reliability, validity, objectivity, & usability.


    Essay items l.jpg

    Essay Items

    • Advantages:

      • Measures higher learning levels (synthesis, evaluation) and is easier to construct than an objective test item

      • Students are less likely to answer an essay question by guessing

      • Require superior study methods

      • Offer students an opportunity to demonstrate their abilities to:

        • Organize knowledge

        • Express opinions

        • Foster creativity


    Essay items30 l.jpg

    Essay Items

    • May limit the sampling of material covered

      • Tends to reduce validity of the test

  • Disadvantages

    • Subjective unreliable nature of scoring

      • “halo effect” – good or bad student’s previous level of performance

      • Written expression

      • Handwriting legibility

      • Grammatical and spelling errors

  • Time Consuming


  • Essay questions31 l.jpg

    Essay Questions

    • Give students a clear idea of the scope & direction intended for the answer

      • Might help to start the question with the description of the required behavior (e.g., compare, analyze)

    • Appropriate language level for students

    • Construct questions that require students to demonstrate a command of background info, but do not simply repeat that info.

    • If question calls for an opinion, be sure that the emphasis is not on the opinion but on the way its presented or argued.

    • Use a larger number of shorter, more specific questions rather than one or two longer questions so that more information can be assessed.


    Essay questions32 l.jpg

    Essay Questions

    • You might

      • Give students a pair of sample answers to a question of the type you will give on the test.

      • Sketch out a rubric (grading scheme) for each question before reading the papers OR randomly select a few to read and make up the grading scheme based on those answers

      • Give students a writing rubric

      • Detach identifying information and use code numbers instead to avoid letting personality factors influence you.

      • After grading all the papers on one item, reread the first few to make sure you maintained consistent standards

      • Be clear to student the extend to which factors other than content (e.g., grammar, handwriting, etc.) will influence the grade.


    Essay questions33 l.jpg

    Essay Questions

    • Tips for constructing Essay Questions

      • Provide reasonable time limits for each question

        • “thinking and writing time”

      • Avoid permitting students a choice of questions

        • Will not necessarily get a representative sample of student achievement. Only be requiring all students to answer all questions can their achievement be compared

      • A definite task should be put forth to the student

        • Critical words: compare, contrast, analyze, evaluate, etc.


    Scoring essay items l.jpg

    Scoring Essay Items

    • Write an outline of the key points (use outline to design a rubric)

    • Determine how many points are to be assigned to the question as a whole and to the various parts within it.

    • If possible, score the test without knowledge of the student’s name

      • Face Sheet

    • Score all of the answers to one question before proceeding to the next question

      • Consistent standard


    Scoring essay exams l.jpg

    Scoring Essay Exams

    • If possible, score each set of answers within the same time frame

    • Handwriting, spelling & Neatness

      • Two separate grades?

        • Mastery of material

        • Other


    Alternative methods of assessment l.jpg

    Alternative Methods of Assessment

    • Research/Term Papers

    • Research Reviews

    • Reports

    • Case Studies

    • Portfolios

    • Projects

    • Performances

    • Peer evaluation

    • Mastery

    • Simulations


    Cheating l.jpg

    Cheating

    • Preventing Cheating

      • Reduce the pressure (multiple evaluations)

      • Make reasonable demands (length/content of exam)

      • Use alternative seating

      • Use alternative forms

      • Be cautious with extra copies


    Using assessment evaluation to improve student learning outcomes l.jpg

    Using Assessment & Evaluation to Improve Student Learning Outcomes

    • Providing feedback to student

    • Closing the assessment & evaluation loop

    • Maximizing student learning


    Questions l.jpg

    Questions?


  • Login