Setting maintaining exam standards
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

SETTING & MAINTAINING EXAM STANDARDS PowerPoint PPT Presentation


  • 86 Views
  • Uploaded on
  • Presentation posted in: General

SETTING & MAINTAINING EXAM STANDARDS. Raja C. Bandaranayake. NORM-REFERENCED Relative Based on peer-performance Varies with each group Cut-off point not related to competence. CRITERION-REFERENCED Absolute Not related to peer performance Standard set prior to exam

Download Presentation

SETTING & MAINTAINING EXAM STANDARDS

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Setting maintaining exam standards

SETTING & MAINTAINING EXAM STANDARDS

Raja C. Bandaranayake


Norm criterion referenced standards

NORM-REFERENCED

Relative

Based on peer-performance

Varies with each group

Cut-off point not related to competence

CRITERION-REFERENCED

Absolute

Not related to peer performance

Standard set prior to exam

Referenced to a definedlevel of performance

NORM- & CRITERION-REFERENCED STANDARDS


Nedelsky 1954 method example

NEDELSKY (1954) METHOD: Example

  • Consider N judges and n MCQ items of 1 in 5 type

  • Judge A identifies 2 options in item 1 as those which a minimally competent examinee should eliminate as incorrect.

  • MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3

  • Similarly, in item 2 he identifies 3 options, giving an MPLA2 = 1/(5-3) = 1/2

  • He repeats this process for each item.

  • The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 + ………….MPLAn

  • Similarly, Judge B’s MPL [MPLB] is determined

  • The MPL for the exam (= cut-off score) is: (MPLA + MPLB + MPLC +….MPLN) / N


Angoff 1971 method example

ANGOFF (1971) METHODExample

  • N judges consider 100 minimally competent examinees taking an MCQ exam of n items.

  • Judge A estimates that, of these examinees, 50 should answer item 1 correctly, 20 item 2 correctly, 70 item 3 correctly, and so on to item n.

  • The MPL for Judge A [MPLA] = (0.5 + 0.2 + 0.7 + . xn) / n X 100 = (say) A%.

  • Similarly, for Judges B, C, D, E, …..N, the MPLs would be B%, C%, D%, E% ……N%, respectively.

  • The MPL (cut-off score) for the exam is: (A% + B% + C% + D% + E% +....N%) / N


Ebel 1972 method example

EBEL (1972) METHODExample

  • Assume that Judge A assigns items in a 200-item MCQ test to the cells of a “relevance-by-difficulty” matrix, as follows.

  • He then estimates the percentage of items in each cell of the matrix that a minimally competent examinee should be able to answer correctly (as indicated within the cell).

  • Each cell also includes the products of these two values.

    • EASYMEDIUMHARD

      ESSENTIAL 15 x 100% = 1500 25 x 80% =2000 10 x 60% = 600

      IMPORTANT 20 x 80% = 1600 40 x 60% =2400 20 x 50% =1000

      ACCEPTABLE 10 x 50% = 500 25 x 40% = 1000 5 x 10% = 50

      QUESTIONABLE 10 x 30% = 300 15 x 20% = 300 5 x 0% = 0


Ebel 1972 method contd example

EBEL (1972) METHOD - contd.Example

  • The MPL for Judge A [MPLA] is then:

  • (1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000 + 50 + 0) / 200 = 56.25 %

  • Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD] …..N [MPLN] are determined.

  • The MPL for the exam (cut-off score) is:

  • (MPLA+MPLB+ MPLc+ MPLD + …..MPLN) / N


Setting maintaining exam standards

PROPOSED EBEL MODIFICATIONEASY MEDIUM HARDESSENT. 6x 100% = 600 12 x 80% = 960 7 x 50% = 350 IMPORT. 12 x 80% = 960 24 x 60% = 1440 19 x 40% = 760 ACCEPT. 5 x 60% = 300 12 x 50% = 600 3 x 10% = 30 MPL: =600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30=6000/100= 60


Setting maintaining exam standards

A

fmax

20

Failure

Rate%

15

B

fmin

10

35

40

45

50

Cut-off score(%)

cmin

cmax

HOFSTEE METHOD


Setting maintaining exam standards

HOFSTEE METHOD

Example

A plot of cut-off scores for a given exam against resulting failure rates is given

cmin = 40%

cmax = 45%

fmin = 10%

fmax = 20%

A = point representing cmin,fmax

B = point representing cmax,fmin

Line AB intersects the curve at a cut-off point of 42.5%

Thus, operational cut-off score = 42.5%


Cut off score for 1 in 5 mcq fracs part 1

CUT-OFF SCORE FOR 1 IN 5 MCQ[FRACS PART 1]

  • Probability of guessing (=1 in 5) = 20%

  • ‘Total ignorance’ score= 20%

  • Maximum possible score=100%

  • Effective range of scores= 20% to 100%

  • Mid-point of this range= 60%

  • Additional factor (as PG exam)= 5%

  • Nominal cut-off score (60%+5%) = 65%


Setting maintaining exam standards

CUT-OFF SCORES: “MARKER QUESTIONS”

1. Comparison of exam scores

Mean score in this exam: 56.7%

Average exam mean score over last 4 years: 59.4%

Thus mean score in this exam is: 2.7% lower

Assuming this candidate group is of same

standard as in last 4 yrs, this exam is: 2.7% harder


Setting maintaining exam standards

CUT-OFF SCORES: “MARKER QUESTIONS” - contd.

2. Comparison of “marker” scores

Mean score in this exam on previously used

questions (N=162): 62.5%

Mean score on same questions when they

were each last used: 60.5%

Thus, when compared with previous

candidates, this group of candidates, on

these items, scored (62.5-60.5)% = 2.0% higher

Thus this group of candidates is: 2.0% betterthan previous groups


Setting maintaining exam standards

CUT-OFF SCORES: “MARKER QUESTIONS” – contd.

3. Estimating examination difficulty

Thus it is expected that their mean score

in this exam would be:2.0%higher

But their mean score in this exam is: 2.7%lower

Thus this exam is really: 4.7%harder


Setting maintaining exam standards

CUT-OFF SCORES: “MARKER QUESTIONS” –contd.

4. Determining cut-off score

The cut-off level for an average exam is:65.0%

Thus the cut-off level for this exam

should be (65 – 4.7)% = 60.3%

Cut-off score = 60.3%


  • Login