- 63 Views
- Uploaded on
- Presentation posted in: General

Advanced Topics in Standard Setting

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Advanced Topics in Standard Setting

- Methodology
- Implementation
- Validity of standard setting

- Students taking standard-based tests must perform in such a way that it can be evident from their test scores that they have achieved at least a minimum level of competency (MLC). This MLC is defined operationally as the cut score.
- The process of recommending the cut score is called standard setting.

- Test Centered
- Examinee Centered
- Item Mapping

- Test Centered:
- Items are presented to the panelists in the same order as they appear on the test
- Panelists predict a target student’s (e.g., a barely proficient) performance on each item in the test

- Very popular in licensure and certification tests
- What is the probability that a barely proficient student will answer the item correctly?

- Cognitively very challenging
- Conceptualize a barely proficient student
- Predict probability of giving correct response to the item

- Panelists tend to underestimate difficulty of an easy item and overestimate difficulty of a hard item

- For each prediction panelist asks, “whether 2/3 of the barely proficient students will be able to answer the test item correctly (yes or no)?”
- Cognitively less challenging
- Used only for MC item tests

- Each item has maximum possible score of 4.
- Scoring rubric: students can get only 0, 1, 2, 3, or 4 points.
- What is the probable score that a barely proficient student will get on the item?

- Examinee Centered:
- Panelists identify a target student’s paper (i.e., student folder) that is consistent with performance level descriptions

- Popular Methods
- Analytical Judgmental Method (2000)
- Used for only CR and writing tests
- Not used for a test that has both MC and CR items

- Body of Work (1994)

- Analytical Judgmental Method (2000)

- Item Mapping:
- Items are presented to panelists in an order item booklet (OIB) based on difficulty level of the items
- Panelist review each item and determine what knowledge, skills, and abilities must be required to answer a given item correctly
- What makes each item progressively more difficult than the previous item in the OIB.

- Bookmark (1995)
- Can be used both MC and CR item tests
- Cognitively less challenging

- Used in NAEP (2005)
- Too expensive: may not be suitable for state assessment programs

- Mixed Method (2006)
- Blend of Angoff and Bookmark standard setting method
- Using the strengths of these two methods
- Not yet operationally implemented
- Experts in standard setting (personal communication) seem to have liked this method.

- Select the method that is appropriate for the assessment program
- Test item type
- Consistency, method used previously in the program
- Prior experience with the method (person implementing)
- Resources available

Implementation

- General Standard Setting Process
- Selection of panelists
- Orientation
- Review test materials
- Review and discuss Performance Level Descriptors (PLDs)
- Round 1 ratings
- Feedback
- Round 2 ratings
- Evaluation

(1) Psychometricians (2) DOE staffs (3) Item writers (4) Subject matter experts

- Content knowledge
- Understands student population
- Knowledge of instructional environment
- Appreciates the consequences of the standards
- Relevant stakeholders (parents, university level educators, etc.)

- How many panelists should we have for a standard setting study?
- 0 (b) 5-6 (c) 7-9 (d) 10-15 (e) 16-40

- 10-15 panelists may be sufficient to set a defensible cut score
- The magnitude of error is influenced by more than the number of participants
- It suggests that precision of the cut score is influenced by other factors.

- Qualification of panelists
- Orientation
- Impact data
- NCLB
- Conceptualization of a barely proficient student
- Thinking about one, who is in my classroom

- How many times should we collect panelists’ ratings?
- 0 (b) 2 (c) 3 (d) 4 (e) 5

- Two rounds of panelists’ ratings may be adequate to get reasonable and defensible cut score
- Third round ratings may not differ much from round two ratings

- Should we provide feedback to the panelists between the rounds?

- If yes, what kind of data?

- We often provide
- Impact data (e.g., if the cut score is 23 out of a possible score point of 40 what % of students in the state will be classified as Proficient?)
- Summary of their round 1 ratings (Panelists’ locations)
- Student’s profile

- Process feedback (panelists’ location) typically has the effect of reducing the standard deviation of cut scores set by panelists
- It is also evident that the effect of feedback diminishes as the number of rounds of feedback and rating increases.

- It will be discussed in the Validity section
- It is a very important component in a standard setting study

Angoff-based Methods

- Web-based standard setting
- Reduce number of items rated by panelists
- Divide the test and the panelists into equivalent groups
- Use only a subset of items
- content and difficulty
- content, discrimination, and difficulty
- 50% items may be adequate

- Bookmark Method
- Use only a subset of items
- Content, item-type, and difficulty
- 70% items may be adequate

- Use only a subset of items

Easiest

Hardest

Cut 1 = avg.(-1.85 and -1.5) = -1.68

Cut 2

Cut 3

Cut 3 = 4

Cut 2 = 3

Cut 1 = 2

Cut 1

Cut 2

Cut 3

- Scaled scores are typically a linear transformation of ability estimates
- Example of a linear transformation:
- (Ability x Slope) + Intercept

- Panelists should be subject matter experts
- 10-15 panelists are adequate for a standard setting study
- Two-round of panelists’ rating may be sufficient to estimate a defensible cut
- A reasonable feedback data should be provided

- When designing a standard setting study, potential influencing factors should be considered
- Streamlined standard setting procedures may be a good consideration for low-stake tests.

Validity of Standard Setting Process

- Assumptions:
- Policy assumption: It claims that the performance standards are appropriate, given the purpose of the decisions
- Operational assumption: It claims that students with scores at or above the cutscores are likely to meet the performance standards, and students with scores below the cutscore are not likely to meet the standard.

- Policy assumption is often evaluated by documenting procedural evidence (e.g.,)
- Purposes of the decision process
- Selection of panelists
- Training of panelists
- Definition of performance standard
- Data collection procedure

- Operational assumption is examined through internal consistency evidence and external criteria
- Internal consistency: Results that are not internally consistent do not justify any conclusions.

- Precision of estimates of the cutscore
- Standard error of the cutscore: If the standard setting study was repeated, to what extent we would be likely to get the same cutscore

- Analysis of item-level data
- Examining performances of students with scores near the cut score
- Comparing performance of two groups of students (one with scores a bit above the cut and the other a bit below the cut)

- Examining performances of students with scores near the cut score. For example, panelists set the proficient cut at 25 out of a possible score of 40.
- For an item, panelists think almost 90% of borderline proficient students should be able to answer the item correctly. If conditional p-value of the item for students who got a score of 25 is much different than 0.90, it implies that the cut score may not be accurately placed.

- Comparing performance of two groups of students (one with scores a bit above the cut and the other a bit below the cut):
- Compare conditional p-values of the items. We would expect that p-values for above-the-cut student group will be higher than p-value for below-the-cut student group.

- Intra-panelist consistency: A measure of how consistently a panelist provides judgments across the items
- Inter-panelist consistency: A measure of how consistently the panelists provide judgments on each item
- An index that measures inter-panelist consistency for an Angoff, Body of work, and Bookmark standard setting methods

- Comparisons to results of other standard setting methods
- Challenges: Different methods are not equally appropriate for a certain type of test

- Judgments by stakeholder groups (e.g., classroom judgment data)
- Challenges: Finding stakeholders who are qualified to make this judgment (e.g., stakeholders may have incomplete understanding about the performance level definition)

- Comparisons involving other assessment methods
- Existing classification data could be used as the basis for checking the appropriateness
- Challenges: Finding an appropriate external criterion

- Standard setting (or setting performance standards) is a judgmental method
- Different methods may set different performance standards for the same test
- This is ultimately a policy decision
- Standard setting procedure needs to be well documented in order to make it defensible.