1 / 14

Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid

Development team. Matt Ohland, ClemsonHal Pomeranz, Deer Run AssociatesLisa Bullard, NC StateCindy Finelli, U MichiganRichard Layton, Rose-HulmanRich Felder, NC StateMisty Loughry, ClemsonDoug Schmucker, Western Kentucky. Motivation for Development. Research PurposesIdentify critical areas o

pauline
Download Presentation

Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. 1 Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid

    2. Development team Matt Ohland, Clemson Hal Pomeranz, Deer Run Associates Lisa Bullard, NC State Cindy Finelli, U Michigan Richard Layton, Rose-Hulman Rich Felder, NC State Misty Loughry, Clemson Doug Schmucker, Western Kentucky

    3. Motivation for Development Research Purposes Identify critical areas of team-member contributions Create instrument for use in academic or work settings Team projects are common in management and engineering courses Teamwork common in work organizations Recruiters of business students are looking for people with good interpersonal skills who can work well on teams The ability to function in teams is part of the accreditation standards in engineering Recruiters of business students are looking for people with good interpersonal skills who can work well on teams The ability to function in teams is part of the accreditation standards in engineering

    4. Peer Evaluations Account for performance and behavior that outside observers do not see Reduce the tendency to hitchhike Help users understand performance criteria Have good predictive validity with various performance criteria

    5. Reducing the complexity 5 broad (second-order) factors of team-member contribution Developed a Behaviorally Anchored Rating Scale Contributing to the teams work Interacting with teammates Keeping the team on track Expecting quality Having task-related knowledge, skills, and abilities

    6. Web-based instrument Benefits for students Quick feedback on team skills Improved anonymity Benefits for faculty Paperwork reduction Quick feedback on dysfunctional teams Automated data processing Customizable Benefits for research - improved validity

    7. Demonstrating the system Request a faculty account at www.catme.org () Email directs you to setup page () For now, use password catme Use the wizard to create a class Create survey within the class Sample student list ftp://www.catme.org/ Go with survey defaults

    8. The student perspective Email issued to students when they are added to the system Email issued when survey opens Take the survey Log in as a1@deer-run.com, b1@... Password is catme View the results the students see Results displayed graphically

    9. After the survey (demo) Completion percentage Reminders to students Releasing results to students Averages by default, raw data available Grade adjustment factor Exceptional conditions

    10. Exceptional Conditions Manipulator (red) Overconfident (yellow) Underconfident (yellow) High performer (green) Low performer (yellow) Personality conflict (red) Clique (red)

    11. How to use the results Pop-ups for comments and conditions Disabling pop-ups Copy / paste to Excel

    12. Quotes from users The results do confirm what I thought about some of the students, and there are a few interesting things - personality conflicts mainly. (faculty user) I enjoyed having a chance to give a fair evaluation of my group and each members contribution and participation. It has only helped mine, [but] I still am not totally comfortable having a survey factored into my grade.

    13. Quotes from users It makes the process of evaluating your team members very easy and effective. These surveys are helpful for the teachers to know the individual's work habits. Not everyone does an equal amount of work and this helps give fair grades. This is the first CATME survey I have done, unfortunately. The first two [hurt my grade]. I have not contributed as much [because my partners] were already friends before our group was created, [and I was excluded].

    14. Persistent problems Social concerns Unwillingness to take time to carefully read items and think about ratings Low variance Average rating perceived as bad Less competent students less aware of their shortcomings

    15. Instrument / NSF Support Web-based instrument found at http://www.catme.org/ This material is based upon work supported by NSF Award 0243254, Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid.

More Related