slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeho PowerPoint Presentation
Download Presentation
Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeho

Loading in 2 Seconds...

play fullscreen
1 / 21

Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeho - PowerPoint PPT Presentation


  • 150 Views
  • Uploaded on

Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeholders. Presented by Denise Sibley Laura Jean Kerr Mississippi Assessment Center Research and Curriculum Unit. Measuring Technical Skill Attainment.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeho' - micheal


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeholders

Presented by

Denise Sibley

Laura Jean Kerr

Mississippi Assessment Center

Research and Curriculum Unit

measuring technical skill attainment
Measuring Technical Skill Attainment
  • Requirement established by Perkins IV
  • Assessments have been developed
  • Standard setting needs to be established
what is standard setting
What is Standard Setting?

The proper following of a prescribed, rational system of rules and procedures that result in determining:

  • cut scores and performance levels
  • what’s tested
  • proper statistical analyses
benefits of standard setting
Benefits of Standard Setting
  • Ensures that decisions are based on high-quality data
  • Ensures that data are combined in a systematic, reproducible, objective, and defensible manner
  • Increases understanding and trust on the part of the stakeholders
choosing a standard setting method
Choosing a Standard-Setting Method
  • Researched standard setting methods
  • Limited literature for CTE assessments – lots of academic literature
  • Method used had to present a strong validity argument for the results
  • Method used had to be suitable for the CTE content
  • Method used had to be straight-forward
item descriptor matching
Item Descriptor Matching
  • Performance level descriptors (PLDs) need to developed
    • Descriptions of the performance levels into which examinees will classified (e.g., Minimal, Basic, Proficient, or Advanced)
  • Participants rate items according to PLDS
  • Items with agreement less than .67 are re-rated
  • Use results to distinguish between the four performance level categories.
prepare descriptions of the performance categories
Prepare Descriptions of the Performance Categories
  • PLDs should be specific to each standard in the curriculum
  • Challenging because there were over 100 curricula
  • Developed Career and Technical Education PLDs
career and technical education plds
Career and Technical Education PLDs
  • Advanced: An advanced student evaluates and integrates transferable academic/workplace knowledge and skills in multiple situations as a productive contributor in the workplace.
  • Proficient: A proficient student acquires and applies academic/workplace knowledge and skills to become a productive contributor in the workplace.
  • Basic: A basic student has limited acquisition and comprehension of academic/ workplace knowledge and skills that are necessary to become a productive contributor in the workplace.
  • Minimal: A minimal student has not developed academic/workplace knowledge and skills that are necessary to become a productive contributor in the workplace.
program description and standards from curricula
Program Description and Standards from Curricula

Included program descriptions and standards from the curricula as a reference for participants to ensure they aligned their ratings with the curriculum

selecting subject matter experts smes
Selecting Subject Matter Experts (SMEs)

Used stratified random sample method for selecting participants–geographic location based on 15 community colleges

Randomly selected teachers from each geographic location by program area

Had to be able to attend standard setting meeting

Approximately 400 subject matter experts participated

materials for standard setting meeting
Materials for Standard-Setting Meeting
  • PLDs
  • Standards from each curriculum framework
  • Program description from each curriculum framework
  • Test item booklet with only field-tested items
    • Item difficulty is used to determine the range for each performance level
  • Sign-in sheets with logins and passwords
  • Bubble sheets for backup if site networks went down
conducting the meetings
Conducting the Meetings
  • Facilitators were trained prior to the meetings to insure that meetings were conducted in a consistent manner
  • Meetings were held across the state over the course of 3 days
  • A Web interface was developed for participants to enter their responses
  • Scheduled participants from each program area to meet on the same day
  • Participants received instructions on using the materials and the Web interface
  • Participants went through test item booklets and matched items to PLDs and also were able to write comments about the items
  • When all participants submitted their responses, they were calculated and any items that didn’t have a .67 agreement were re-rated.
  • Participants completed a survey after they completed the meeting
using technology
Using Technology
  • Conduct Standard Setting over long distances
  • Coordinate Standard Setting for the best use of Subject Matter Experts time
  • Keep up with ever-changing needs of Career and Technical Education
  • Organize and compile data
consistency in standard setting
Consistency in Standard Setting
  • Consistency is the justification for subjective limitations
    • Training of Subject Matter Experts
    • Standard Setting environment
    • Policies and procedures
  • Consistency contributes to quality data
survey comments
Survey Comments
  • Communication
    • “The PowerPoint would have been helpful if I had seen it before today.”
    • “I did not fully understand the descriptors. I probably over estimated the students' ability.”
    • “The performance level indicators are open to many interpretations.”
  • Lesson learned:
    • Communicate with SMEs early and often.
survey comments17
Survey Comments
  • Meeting Environment
    • “It was the room and atmosphere where the training was held. It was in a room off the cafeteria. Which means...guess what...too much traffic. We could hear the phone conversations next door from an office, when the phone was ringing and etc.”
    • “Could use a break for the 200 questions and refreshments.”
    • “People who arrive late should not be able to enter.”
  • Lesson Learned:
    • Every aspect (even the smallest detail) of the testing environment should be taken into consideration.
survey comments18
Survey Comments
  • Technology
    • “I would like to see the questions online with the performance descriptors. I believe the process would go a lot faster and would allow for easier comments.”
    • “Website does not allow you to go back and alter responses on previous pages in event of mistakes.”
    • “Love doing this on computer!”
  • Lesson learned:
    • Incorporated these comments into development of new interface
survey comments19
Survey Comments
  • Trust and Understanding
    • “I appreciate being able to give feedback on the questions. This is the first time I have been invited to do so.”
    • “I think that this is a step in the right direction for standardized testing.”
    • “Good program...would like to know more about how the process works with our peers, etc.”
  • Keeping stakeholders in the “loop” fosters trust
final steps
Final Steps
  • Compile data
  • Provide feedback to stakeholders
  • Document process
  • Improve process
  • Publish results
contact information
Contact Information

Denise Sibley

Interim Coordinator of Assessment and Accountability

denise.sibley@rcu.msstate.edu

Laura Jean Kerr

Assessment Specialist

laura.kerr@rcu.msstate.edu