Evaluation of Student Learning

Evaluation of Student Learning PowerPoint PPT Presentation


  • 264 Views
  • Updated On :
  • Presentation posted in: General

Download Presentation

Evaluation of Student Learning

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


1. Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional Development Fall 2005 Dr. Kristi Roberson-Scott Lecturing and other classroom activities are often seen as the enjoyable aspect of teaching, while evaluating student work and assigning grades are probably less appealing. However, constructing tests is an important part part of most courses. Tests help instructors and students keep track of the kind and amount of learning that is taking place. Tests given at the end of a unit, mid-semester or end of semester. Virtually everyone who has thought carefully about the question of assessing quality in higher education agrees that “value added” is the only valid approach. Excellence & quality should be determined by the degree to which we, as a college, develop the abilities of our students. By “value added” – value that is added to students’ capabilities and knowledge as a results of there education here at RSCC. Measuring such value requires assessing what students know and can do as they begin college, assessing them during college and again at end, after college grad. Value added is thus the difference between the measures of students’ achievement as they enter college and measures of achievement when they complete college. Value added is the difference a college makes in their education. Lecturing and other classroom activities are often seen as the enjoyable aspect of teaching, while evaluating student work and assigning grades are probably less appealing. However, constructing tests is an important part part of most courses. Tests help instructors and students keep track of the kind and amount of learning that is taking place. Tests given at the end of a unit, mid-semester or end of semester. Virtually everyone who has thought carefully about the question of assessing quality in higher education agrees that “value added” is the only valid approach. Excellence & quality should be determined by the degree to which we, as a college, develop the abilities of our students. By “value added” – value that is added to students’ capabilities and knowledge as a results of there education here at RSCC. Measuring such value requires assessing what students know and can do as they begin college, assessing them during college and again at end, after college grad. Value added is thus the difference between the measures of students’ achievement as they enter college and measures of achievement when they complete college. Value added is the difference a college makes in their education.

2. Guiding Principles for Evaluation Evaluation should relate directly to instructional objectives Each evaluation activity should be designed to promote student growth The actual activity should be useful practice in itself Feedback should be useable by the student Multiple evaluation strategies should be provided to master achievement of X objective/competency Student should clearly understand the methods of evaluation for X test or activity

3. Questions to Ask yourself in Designing a Test What objectives will (should) I be testing? What types of items will be included in the test? How long will the test be in terms of time and number of items? How much will each objective be worth in terms of weighting and number of items?

4. Tests as Diagnostic Tools Students demonstrate learning Instructor effectiveness – modify teaching strategies or activities Assignment of letter grades

5. Different Types of Tests & Learning Paper & Pencil/WebCT Testing Limited Choice Questions (MC, T/F, Matching) Open-Ended Questions (Short Answer, Essay) Performance Testing Acquisition of skills that can be demonstrated through action (e.g., music, nursing, etc.)

6. Planning a Test First step: Outline learning objectives or major concepts to be covered by the test Test should be representative of objectives and material covered Major student complaint: Tests don’t fairly cover the material that was supposed to be canvassed on the test.

7. Planning a Test Second Step: Create a test blueprint Third Step: Create questions based on blueprint Match the question type with the appropriate level of learning Fourth Step: For each check on the blueprint, jot down (might use 3x5 cards) 3-4 alternative question on ideas and item types which will get at the same objective Fifth Step: Organize questions and/or ideas by item types Level of learning objective Content coverage Practice and reward of writing and reading skills Reward of creative and divergent thinking Feedback to instructor and student Length Size of class Reliability in grading Exam construction and grading time Test resuablity Prevention of cheatingLevel of learning objective Content coverage Practice and reward of writing and reading skills Reward of creative and divergent thinking Feedback to instructor and student Length Size of class Reliability in grading Exam construction and grading time Test resuablity Prevention of cheating

8. Planning a Test Sixth Step: Eliminate similar questions Seventh Step: Walk away from this for a couple of days Eighth Step: Reread all of the items – try doing this from the standpoint of a student

9. Planning a Test Ninth Step: Organize questions logically Tenth Step: Time yourself actually taking the test and then multiply that by about 4 depending on the level of students Eleventh Step: Analyze the results (item analyses)

10. Translating Course Objectives/Competencies into Test Items Syllabus Specification table- what was taught/weight areas to be tested Creating a Test Blueprint (see handout) Blueprint- this is the test plan, i.e., which questions test what concept Plotting the objectives/competencies against some hierarchy representing levels of cognitive difficulty or depth of processing

11. Thinking Skills What level of learning corresponds to the course content Bloom’s Taxonomy of Educational Objectives Knowledge (see handout) Comprehension Application Analysis Synthesis Evaluation

12. Practical Considerations Representative sample of the course content not random– purposeful based on blueprint Representative sample of skill or cognitive levels across content Analyze results by level AND content area Bullet # 3: If student are getting all of the lower level questions but missing higher level, then you’re teaching strategies may not be effective to master upper level cognitive skills Bullet # 3: If student are getting all of the lower level questions but missing higher level, then you’re teaching strategies may not be effective to master upper level cognitive skills

13. Question Arrangement on a Test Group by question type Common instructions will save reading time Limit the number of times students have to change frame of reference Patterns on test must be logical Arrange from a content standpoint Keep similar concepts together Group by difficulty (easy to hard) Don’t mix questions about different plays by the same author, different paintings by the same painter All questions about quadratic equations in one place Easy to hard – get them warmed up, don’t get them too defeated before they get to the easy ones, don’t have them run out of time before getting to the easy ones – Example of a Chemistry Exams Don’t mix questions about different plays by the same author, different paintings by the same painter All questions about quadratic equations in one place Easy to hard – get them warmed up, don’t get them too defeated before they get to the easy ones, don’t have them run out of time before getting to the easy ones – Example of a Chemistry Exams

14. Selecting the Right Type of evaluation How do you know what type of question to use and when? It depends on the skill you are testing. Evaluation should always match as closely as possible the actual activity you’re teaching. Examples: Teaching Speech, should evaluate an oral speech If testing ability to write in Spanish, better give an essay. Testing reading –MC, TF Wouldn’t use MC to test creative writing

15. Question Types verses Cognitive Levels of Learning

16. Constructing the Test Types of Test Questions: Multiple-Choice Items True-False Items Matching Items Fill-In, Completion or Short-Answer Items Essay Questions T/F best suited for testing 3 kinds of information – probably the most widely used and most criticized Advantages: Easy –it’s difficult to discriminate between those that know the material and those who don’t know the material 50-50 chance of answering correctly – actually, the student’s chances are greater thatn 50-50 because instructors tend to include from 2/3 to ¾ true statements and likewise students tend to guess true more than false. T/F best suited for testing 3 kinds of information – probably the most widely used and most criticized Advantages: Easy –it’s difficult to discriminate between those that know the material and those who don’t know the material 50-50 chance of answering correctly – actually, the student’s chances are greater thatn 50-50 because instructors tend to include from 2/3 to ¾ true statements and likewise students tend to guess true more than false.

17. Multiple Choice Items Advantages: Extremely versatile-can measure the higher level mental processes (application, analysis, synthesis and evaluation) A compromise between a short answer/essay and T/F item Can cover a wide range of content can be sampled by one test Disadvantages Difficult to construct plausible alternative responses Compromise Guessing is markedly reduced- In a ten item, four alternative, MC test, the probability of obtaining a score of 7 (passing) is 1 in 1,000. Usually the first couple of distractors are easy to construct but the last 1 or 2 is difficult – time consuming taskCompromise Guessing is markedly reduced- In a ten item, four alternative, MC test, the probability of obtaining a score of 7 (passing) is 1 in 1,000. Usually the first couple of distractors are easy to construct but the last 1 or 2 is difficult – time consuming task

18. Types of Multiple Choice Items Four Basic Types Question Type Incomplete Statement Type Right Answer Type Best Answer Type Which Type is Best? Question Type vs. Incomplete Statement Right Answer vs. Best Answer Type Question Type: Stem is stated in form of a question Incomplete: This is where the student is required to correctly identify the remaining part of the incomplete sentence Question form is generally better when compared to incomplete statement for the following reasons: The purpose of the stem is to confront the student with some problem or situation. The question type forces the test writer to clearly stat the problem. Good incomplete items can be difficult to construct. A litmus test for a good stem is whether or not a student could answer the problem without having to read the alternatives first. This is NOT always the case with an incomplete statement. Right Answer- Identify the one right answer among other that are wrong Best Answer – select best one others are plausible Right Answer vs. Best Answer Advantages to both types .. The construction of one right answer less time consuming. However, the best answer item most had the edge because it requires students to use judgment, reasoning and other types of understanding. Recommendation: Most experts recommend that test writers use the question and best answer types. In the test directions, it should be explicitly pointed out that the student is to choose the best answer. Should be indented five spaces from the stem and stem and responses separated by a blank line Question Type: Stem is stated in form of a question Incomplete: This is where the student is required to correctly identify the remaining part of the incomplete sentence Question form is generally better when compared to incomplete statement for the following reasons: The purpose of the stem is to confront the student with some problem or situation. The question type forces the test writer to clearly stat the problem. Good incomplete items can be difficult to construct. A litmus test for a good stem is whether or not a student could answer the problem without having to read the alternatives first. This is NOT always the case with an incomplete statement. Right Answer- Identify the one right answer among other that are wrong Best Answer – select best one others are plausible Right Answer vs. Best Answer Advantages to both types .. The construction of one right answer less time consuming. However, the best answer item most had the edge because it requires students to use judgment, reasoning and other types of understanding. Recommendation: Most experts recommend that test writers use the question and best answer types. In the test directions, it should be explicitly pointed out that the student is to choose the best answer. Should be indented five spaces from the stem and stem and responses separated by a blank line

19. Multiple Choice Items Writing the stem first: Be sure the stem asks a clear question Stems phrased as questions are usually easier to write Stems should not contain a lot of irrelevant info. Appropriate reading level/terms Be sure the stem is grammatically correct Avoid negatively stated stems

20. Multiple Choice Items Writing the correct response Use same terms/reading level Avoid too many qualifiers Assign a random position in the answer sequence Read the stem and correct response together Generate the distractors/alternative responses

21. Multiple Choice Items Other Tips for Constructing MC Items: Items should have 3-4 alternatives. Stem should present a single, clearly formulated problem Simple, understandable, exclude extraneous words from both stem and alternatives Include in the stem any word that are repeated in each response Avoid all of the above (can answer based on partial information) Avoid none of the above Alternatives: The best number of answers (use 3 alternatives) is four or five. Statistical studies have shown that there is a rapid increase in item reliability from two to four answers and only slight increase between four and five. To regulate None of the above - -used with computational problems – can’t estimate the answer.. Recommendation “none of these” To regulate..Alternatives: The best number of answers (use 3 alternatives) is four or five. Statistical studies have shown that there is a rapid increase in item reliability from two to four answers and only slight increase between four and five. To regulate None of the above - -used with computational problems – can’t estimate the answer.. Recommendation “none of these” To regulate..

22. Multiple Choice Items Alternative responses/distractors should be plausible and as homogeneous as possible Response alternatives should not overlap Two synonymous terms (arithmetic average/mean) Avoid double negatives None of the following are part of the brain except which one? Emphasize negative wording Each item should be independent of other items in the test Information in the stem of one item should NOT help answer another item. Same type of body part: Eye Same type of body part: Eye

23. True-False Test Items Best suited for testing 3 kinds of info.: Knowledge level learning Understanding of misconceptions When there are two logical responses Advantages: Sample a large amount of learning per unit of student testing time Disadvantages: Tends to be very easy 50-50 chance of guessing Tends to be low in reliability

24. Tips for Constructing True/False Items Tips for constructing True-False Items Avoid double negatives Avoid long or complex sentences Specific determiners (always, never, only, etc.) should be used with caution Include only one central idea in each statement Avoid emphasizing the trivial Exact quantitative (two, three, four) language is better than qualitative (some, few, many) Avoid a pattern of answers

25. Objective Test Item Analyses Evaluating the Effectiveness of Items.. Why? Scientific way to improve the quality of tests and test items Identify poorly written items which mislead students Identify areas (competencies) of difficulty Item analyses provided info. on: Item difficulty Item discrimination Effectiveness of alternatives in MC Tests

26. Short-Answer Items Two Types: (Question and Incomplete Statement) Advantages: Easy to construct Excellent format for measuring who, what, when, and where info. Guessing in minimized Student must know the material- rather than simply recognize the answer Disadvantages: Grading can be time consuming More than one answer can be correct Incomplete statement: The part of the brain that controls speech and language is called the_____________? Question Example: What is a Type I error? (term that describes the error that occurs when the null HO is rejected but in fact it is trueIncomplete statement: The part of the brain that controls speech and language is called the_____________? Question Example: What is a Type I error? (term that describes the error that occurs when the null HO is rejected but in fact it is true

27. Short Answer Items Tips for Constructing Short Answer Items Better to supply the term and require a definition For numerical answers, indicate the degree of precision expected and the units in which they are to be expressed. Use direct questions rather than incomplete statements Try to phrase items so that there is only one possible correct response When incomplete statements are used, do not use more than one blank within an item. Spelling Errors – should not be taken into consideration unless they make it impossible to discern whether the student actually knows the correct answer. Spelling Errors – should not be taken into consideration unless they make it impossible to discern whether the student actually knows the correct answer.

28. Essay Questions Types of Essay Questions Extended Response Question Great deal of latitude on how to respond to a question. Example: Discuss essay and multiple-choice type tests. Restricted Response Question More specific, easier to score, improved reliability and validity Example: Compare and contrast the relative advantages of disadvantages of essay and multiple choice tests with respect to: reliability, validity, objectivity, & usability.

29. Essay Items Advantages: Measures higher learning levels (synthesis, evaluation) and is easier to construct than an objective test item Students are less likely to answer an essay question by guessing Require superior study methods Offer students an opportunity to demonstrate their abilities to: Organize knowledge Express opinions Foster creativity Better study skills- Students preparing for this type of test are inclined to outline material to draw cause and effect relationships, summarize material and make inferences Better study skills- Students preparing for this type of test are inclined to outline material to draw cause and effect relationships, summarize material and make inferences

30. Essay Items May limit the sampling of material covered Tends to reduce validity of the test Disadvantages Subjective unreliable nature of scoring “halo effect” – good or bad student’s previous level of performance Written expression Handwriting legibility Grammatical and spelling errors Time Consuming Only a few questions can be asked on an essay test, there may be limited sampling and material covered. This wiOnly a few questions can be asked on an essay test, there may be limited sampling and material covered. This wi

31. Essay Questions Give students a clear idea of the scope & direction intended for the answer Might help to start the question with the description of the required behavior (e.g., compare, analyze) Appropriate language level for students Construct questions that require students to demonstrate a command of background info, but do not simply repeat that info. If question calls for an opinion, be sure that the emphasis is not on the opinion but on the way its presented or argued. Use a larger number of shorter, more specific questions rather than one or two longer questions so that more information can be assessed. Metamorphosis Metamorphosis

32. Essay Questions You might Give students a pair of sample answers to a question of the type you will give on the test. Sketch out a rubric (grading scheme) for each question before reading the papers OR randomly select a few to read and make up the grading scheme based on those answers Give students a writing rubric Detach identifying information and use code numbers instead to avoid letting personality factors influence you. After grading all the papers on one item, reread the first few to make sure you maintained consistent standards Be clear to student the extend to which factors other than content (e.g., grammar, handwriting, etc.) will influence the grade. Example of a Writing Rubric Rereading – harder at first –lighten up Example of a Writing Rubric Rereading – harder at first –lighten up

33. Essay Questions Tips for constructing Essay Questions Provide reasonable time limits for each question “thinking and writing time” Avoid permitting students a choice of questions Will not necessarily get a representative sample of student achievement. Only be requiring all students to answer all questions can their achievement be compared A definite task should be put forth to the student Critical words: compare, contrast, analyze, evaluate, etc.

34. Scoring Essay Items Write an outline of the key points (use outline to design a rubric) Determine how many points are to be assigned to the question as a whole and to the various parts within it. If possible, score the test without knowledge of the student’s name Face Sheet Score all of the answers to one question before proceeding to the next question Consistent standard The objective test play an prominent role in (Face sheet for student’s name that can be folded back – eliminate both positive and negative halo effects)The objective test play an prominent role in (Face sheet for student’s name that can be folded back – eliminate both positive and negative halo effects)

35. Scoring Essay Exams If possible, score each set of answers within the same time frame Handwriting, spelling & Neatness Two separate grades? Mastery of material Other Some inconsistency in scoring of essay questions, result from the emotions and attitudes of the instructor scoring the asnwer Some inconsistency in scoring of essay questions, result from the emotions and attitudes of the instructor scoring the asnwer

36. Alternative Methods of Assessment Research/Term Papers Research Reviews Reports Case Studies Portfolios Projects Performances Peer evaluation Mastery Simulations

37. Cheating Preventing Cheating Reduce the pressure (multiple evaluations) Make reasonable demands (length/content of exam) Use alternative seating Use alternative forms Be cautious with extra copies

38. Using Assessment & Evaluation to Improve Student Learning Outcomes Providing feedback to student Closing the assessment & evaluation loop Maximizing student learning

39. Questions?

  • Login