solving classification problems for symptom validity tests with mixed groups validation n.
Download
Skip this Video
Download Presentation
Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation

Loading in 2 Seconds...

play fullscreen
1 / 181

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation - PowerPoint PPT Presentation


  • 287 Views
  • Uploaded on

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation. Richard Frederick, Ph.D., ABPP (Forensic) US Medical Center for Federal Prisoners Springfield, Missouri. I am not a neuropsychologist. My view of brain. Your view of brain. My board certifications:.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation' - mohammad-gamble


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
solving classification problems for symptom validity tests with mixed groups validation

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation

Richard Frederick, Ph.D., ABPP (Forensic)

US Medical Center for Federal Prisoners

Springfield, Missouri

slide2

I am not a neuropsychologist.

My view of brain

Your view of brain

my board certifications

My board certifications:

Forensic Psychology

American Board of Professional Psychology

Assessment Psychology

American Board of Assessment Psychology

my professional goal

My professional goal:

Use tests properly in forensic psychological assessments

slide5

Goals of workshop

Participants in this workshop will be able

to employ Excel graphing methods:

--to evaluate classification characteristics of

symptom validity tests

--to adapt symptom validity test scores to their

individual, local, base rates

--to combine information from local base rate

and multiple symptom validity tests

something is terribly wrong
Something is terribly wrong
  • The SIRS has sensitivity = .485 and specificity = .995.
  • The SIRS was administered to 131 criminal defendants
  • who were strongly suspected of feigned psychopathology.
  • 68% of them were categorized as feigning by the SIRS
slide8

What is a classification test?

A structured routine for determining

which individuals belong to which

of two groups.

slide9

There are two groups.

  • (2) It’s not easy to determine which
  • group an individual belongs to
  • without the help of the test.
slide11

The distributions represent our

estimations of how the populations of

the two groups score on the test.

We generally estimate the population

distributions by sampling. We notice

that the populations have two separate,

but overlapping distributions. The extent

of the overlap is of concern to us.

slide12

Questions that must be addressed in

  • research before we can continue:
  • Are there really two separate groups?
  • Can we effectively represent the
  • population distributions by sampling?
slide14

What we notice next.

The mean separation between the

groups is 10 points.

Persons in Population A have a mean

score that is 10 points below persons in

Population B.

The sd for each population is the same. The

mean separation between groups is one sd.

slide15

When researchers talk about mean

separation, they often refer to effect size.

Often, Cohen’s d is the statistic used to

refer to standardized mean separation.

Here, Cohen’s d = 1. This is often referred

to as a large, or very large, effect size.

slide16

Mean separation = 0

Making tests often means finding those characteristics that best separate the distributions of the two groups.

Two distributions of gender

with respect to:

Intelligence

slide17

Moderately large mean separation

Two distributions of gender

with respect to:

Longevity

slide18

Large mean separation

Two distributions of gender

with respect to:

Hair Length

slide19

Very large mean separation

Two distributions of gender

with respect to:

Body Mass

slide21

Summary:

  • We have two groups.
  • We have a test for which the two
  • groups score differentially.
  • (3) The differences in mean scores
  • represents a very large effect.
slide24

More commonly, researchers report

Sensitivity and Specificity. These terms

are common, but not most helpful.

We are going to use the terms:

True Positive Rate (TPR) and

False Positive Rate (FPR).

TPR = Sensitivity

FPR = 1 - Specificity

slide25

What are TPR and FPR?

TPR is the proportion of individuals who do have the condition who generate positive scores. TPR is the rate of scores are beyond the cut in the direction that indicates the presence of the condition.

FPR is the proportion of individuals who do NOT have the condition who generate positive scores. FPR is the rate of scores beyond the cut in the direction that indicates the presence of the condition.

slide26

The green line represents

the cut score. Scores to the

LEFT of the line are classified

NEGATIVE. Scores to right

are classified POSITIVE.

Have nots

Haves

Here, the False Positive Rate is 92.4%.

The True Positive Rate is 100%.

As we move the line to the right, both

rates DECREASE.

slide30

To totally eliminate

false positives, we have

to be willing to identify

almost no one as a

positive.

slide31

TPR = True Positives/Haves

FPR = False Positives/Have Nots

slide32

Haves

Have nots

slide33

A positive score will be one that is

associated with Population A

membership. If we set a point at which

a score will be used to say, “This score

represents Population A,” such a score

will be referred to as a “positive score.”

A positive score can be a true positive

or a false positive: unknown to us.

slide34

The True Positive Rate is the proportion of

Population A members who generate a

positive score.

In our figure, the point at which we

begin to identify “positive scores” is at 50, the

mean of population A. Scores at or below 50

are called positive, and a person who

generates a positive score is classified as a

Population A member.

slide35

We can pick any value to be our “cut score,” but

it’s hard to pick one that doesn’t result in some

Population B members producing “positive

scores.”

In our figure, 50% of the Population A members

have scores at 50 or below. This is the True

Positive Rate. TPR = .50.

In our figure, 16% of the Population B members

have scores at 50 or below. This it the False

Positive Rate. FPR = .16.

slide37

We note that it is not the test that has

a certain TPR and FPR.

It is the chosen test score that has a

certain TPR and FPR.

A different test score will almost

certainly have different TPR and FPR.

slide38
Overcoming limiting factors of “known groups” validation in determining test score sensitivity and specificity
slide39

We think of a test as a way to characterize a dependency.

As you have more of X, you have more of Y.

Y depends on X.

X predicts Y.

X is some construct. Y is some test score.

There is a relationship that we wish to characterize and

quantify.

slide40

Let’s consider feigning.

As you are more likely to feign, you are more likely to

engage in certain behavior.

This behavior might be “providing answers to items on a

test” at a certain rate.

You might choose more items, you might choose fewer

items than “normals.”

slide41

We develop the idea that we can identify individuals who

respond at a certain rate as feigners, and we decide to

make a decision point about when we call test takers

feigners and when we don’t.

We call that decision point a cut score.

We call test scores at or beyond the cut score:

positive scores

Some positive scores are correct: true positives

Some positive scores are incorrect: false positives

slide42

If our test is any good, and if the relationship between

X and Y is strong, then our rate of true positives is much

higher than our rate of false positives.

Let’s skip to the end. We are now using the test in our

clinic.

We look over our results. We see a number of “positive

scores.”

We know that those “positive scores” are some unknown

mixture of “true positives” and “false positives.”

We’d like to know what that ratio of that mixture is.

slide43

Here’s how we do it:

First, we estimate what the true positive rate of the cut

score is.

Then, we estimate what the false positive rate of the

cut score is.

Then, we figure out what percentage of people in our

sample are feigning.

Then we can get the ratio of the mixture of our true

positive and false positives in all the positive scores in our

clinic. (We call this positive predictive power.)

slide44

Getting TPR and FPR:

We depend on researchers to tell us what the estimates

of true positive rate and false positive rate are.

They usually do this through a process called

“criterion groups validation.”

People with more confidence than might be called

for refer to this process as “known groups validation.”

slide45

The process is seemingly straightforward.

Identify two groups. One group has the condition.

All the positives in this group are “true positives.”

One group doesn’t have the condition. All the positives

in this group are “false positives.”

The rate of “true positives” is the sensitivity of the test.

TPR = sensitivity.

The rate of “false positives” is the non-specificity of

the test. FPR = 1 – specificity.

slide46

There are many problems with this process, but let’s

focus on the main two.

Problem 1

In Study 1, for a given cut score, researchers report the

TPR is .67 and the FPR is .12.

In Study 2, for the same cut score, researchers report

TPR = .58 and FPR = .09.

Which values do you use?

slide47

Problem 2:

In Study 1, for a given cut score, researchers report the

TPR is .67 and the FPR is .12.

In Study 2, for a different cut score, researchers report

TPR = .58 and FPR = .09.

Which cut score do you use?

test results
Test results

TRUTH

TEST

we say for our test true positive rate 49 100 49 sens 49 false positive rate 1 100 1 specificity 99
We say for our test:True positive rate = 49/100 = 49% [sens = 49%]False positive rate = 1/100 = 1% [specificity = 99%]

TRUTH

TEST

slide54

Because God does not whisper to us anymore,we take this test, our best test, and we say, “This is the best we can do.” Let’s call it our Gold Standard.We will now make criterion groups with this test,and we will call the groups “Known Groups.”We will then validate tests, based on these Known Groups.

slide55
We say for our test:True positive rate = 49/100 = 49% [sensitivity = 49%]False positive rate = 1/100 = 1% [specificity = 99%]

TRUTH

TEST

slide58

Let’s validate a new test, which just happens to be a perfect test. What test diagnostic efficiencies will we assign our new, perfect, test?

“KNOWN” GROUPS

PERFECT TEST

slide59

Let’s validate a new test, which just happens to be a perfect test. What test diagnostic efficiencies will we assign our new, perfect, test?

“KNOWN” GROUPS

Our belief that

we can make

perfect criterion

groups from

imperfect criteria

has led us to

misunderstand

tremendously

what we are

doing.

PERFECT TEST

TPR = 49/50 = 98%, FPR = 51/150 = 34%

slide68

REMINDER: Here is what we are working on—figuring out

which positives in our clinic are true positives.

First, we estimate what the true positive rate of the cut

score is. Then, we estimate what the false positive rate of

the cut score is.

Let’s do that part now.

Then, we figure out what percentage of people in our

sample are feigning.

Then we can get the ratio of the mixture of our true

positive and false positives in all the positive scores in our

clinic.

slide73

When BR = 0, 10% of scores

positive, all false positives

slide74

When we

say FPR = .16

and TPR = .50,

what we’re

saying is that,

no matter

what samples

we test, we

expect to

see no fewer

than 16%

positive scores

and no more

than 50%

positive

scores.

Movement along this line from left to right

represents increasing rate of Population A and

increasing rate of positive scores.

slide75

FPR = .10

TPR = .80

slide81

TOMM

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093

slide82

For any

imperfect

test,

PPP ranges

from 0 to 1

as base rate

ranges from

0 to 1

NPP ranges

from 0 to 1

as base rate

ranges from

1 to 0

NPP

PPP

using mgv to estimate test diagnostic efficiencies of the reliable digit span

Using MGV to estimate test diagnostic efficiencies of the Reliable Digit Span

Laurie Ragatz, PhD

Richard Frederick, PhD

what is reliable digit span
What is Reliable Digit Span?

RDS is a symptom validity measure for Digit Span. The value of RDS is derived by adding longest strings of two trials passed for both

forward and backward Digit Span.

Researched cut scores include 5 or lower, 6 or lower, 7 or lower, or 8 or lower.

reliable digit span example
Reliable Digit Span Example:

Directions: Examinee recalls numbers in the same order they were provided by the examiner

Directions: Examinee recalls numbers in the reverse order they were provided by the examiner

Reliable Digit Span: 4 + 3 =7

slide86

(1) We found all available articles dealing

with RDS and identified the cut scores

investigated. We included simulator studies.

(2) Based on the authors’ decision about

criterion group membership, we

calculated the overall base rate of

malingering in the study.

(3) We observed the overall rate of

positive scores in the study at the

identified cut score.

slide87

(4) We did not include any data for persons

with mental retardation. The rate of

positive scores among persons with mental

retardation was exceedingly high for all cut

scores.

slide88

Criterion group

Test outcome

We have 63 malingerers in a sample of 203. BR = 63/203 = 0.31.

We have 57 positive scores. Proportion positive scores (PPS) is 57/203 = .28. For this study, we plot (BR, PPS) = (.31, .28)

x = .31, y = .28. Our n for WLS = 203.

rds 5 or lower
RDS = 5 or lower

Using weighted least squares regression (with N as the weight),

we regressed Proportion Positive Scores (PPS) on Base Rate (BR)

to generate the Proportion Positive Score Line.

We obtained y-intercept of -.015 (all negative values are

truncated to 0), and slope of .265.

slide90

RDS = 5 or lower

put these data in WLS

to obtain regression

line characteristics

scatterplot

slide92

RDS = 6 or lower

y-intercept = .015, slope = .419

slide94

RDS = 7 or lower

y-intercept = .187, slope = .39

slide96

RDS = 8 or lower

y-intercept = .236, slope = .824

slide98

As we move

from a cut score

of 5 or lower to 6

or lower, we

obtain

substantial

improvement in

TPR estimate

with little cost

in FPR increase.

slide99

Our choice for best cut score for RDS

RDS: 6 or lower, FPR = .015, TPR = .434

slide100

By using WLS regression, we can obtain standard errors of

our estimates of FPR and TPR.

So, new researchers can test hypotheses about parametric

values of FPR and TPR.

slide101
Overcoming limiting factors of “known groups” validation in determining test score sensitivity and specificity
slide102

Summary:

  • The TVS and MGV allow powerful research into existing published data sets. Summary data are used.
  • Understanding of parametric values of TPR and FPR is facilitated when researchers publish results on a variety of cut scores that should be considered. A frequency
  • distribution would be ideal, for example,
slide103

Combining studies in this way allows us to generate

  • stable values of TPR and FPR with SE’s so that new research
  • can test those values.
  • Researchers should focus on the basis for estimating BR’s
  • in their research groups. All research estimating FPR and TPR
  • is vulnerable to error when the purity of research groups is
  • overestimated. Working towards a reliable estimate of
  • mixed group base rate will facilitate better validation studies.
slide104
Reliably estimate local base rates of feigning for proper allocation of sensitivity and specificity information
slide105

How can the Test Validation Summary help me

  • determine my local BR?
  • Get the best estimate of the test FPR and TPR
  • for a certain test score.
  • 2. Find the proportion of test scores in your
  • sample that are positive scores.
slide108

From a sample, observe

rate of positive scores.

Use TVS to estimate

condition BR in that

sample, PPP and NPP

for that BR.

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077

slide109

TOMM

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093

slide110

Beth A. Caillouet, Bernice A. Marcopulos, Jesse G. Brand,

Julie Ann Kent, & Richard I. Frederick

Question: What are the BRs of malingering in the two samples?

slide111

Question: What are the BRs of malingering in the two samples?

Information needed:

Estimates of TOMM FPR and TPR. From TOMM TVS, we

get FPR = .056, TPR = 742.

Sample 1: Secondary gain present. Proportion positive

scores = 55/220 = .25.

Sample 2: Secondary gain absent. Proportion positive

scores = 34/299 = .11.

Use TOMM TVS to estimate BR of each sample.

slide112

When PPS = .25,

BR = .28.

When PPS = .11,

BR = .08.

defensibly choose symptom validity cut scores that are ideally suited for their local base rates
Defensibly choose symptom validity cut scores that are ideally suited for their local base rates
slide117

TPR = .93 FPR = .17

BR malingering = 35%, N = 86

slide118

TPR = .93 FPR = .17

BR malingering = 35%, N = 86

slide119

TPR = .93 FPR = .17

BR malingering = 35%, N = 86

slide120

TPR = .93 FPR = .17

BR malingering = 35%, N = 86

slide121

TPR = .93 FPR = 10/56 = .18

BR malingering = 35%, N = 86

slide122

TPR = 28/30 = .93 FPR = 10/56 = .18

PPP = 28/38 = .737 NPP = .958

BR malingering = .35

NPP PPP 1 – FPR TPR

slide123

TPR = 28/30 = .93 FPR = 10/56 = .18

PPP = 28/131 = .213 NPP = .996

BR malingering = .05

NPP PPP 1 – FPR TPR

slide124

Test validation

summary for

M-FAST cut

score

recommended

by test manual.

PPP does not even reach 50%

correct decisions until BR > .16

M-FAST > 5

FPR = .17

TPR = .93

At recommended cut score FPR very high

slide125

At BR = .05, PPP does not exceed

.50 until cut score adjusted to

> 9 on M-FAST

slide126

Combining information from local base

rate and multiple symptom validity tests

You can get estimates of PPP and NPP

for the sample you work with—IF you

can reliably estimate the BR.

slide127

737 defendants were administered:

Rey 15 Item Memory Test (RMT)—memorize and

reproduce 15 items—very easy test.

Score is items reproduced (0 to 15)

Word Recognition Test (WRT)—memorize 15 words,

identify those 15 and correctly reject 15 from a list of 30.

Score is number of hits and correct rejections (0 to 30)

slide128

RMT validating

using MGV

with clinical

probability

judgments.

FPR = .025

TPR = .574

Frederick & Bowden,

2009

slide129

RMT < 9

FPR = .025

TPR = .574

We found 726 defendants who completed BOTH RMT and WRT.

81/726 failed the RMT= .111 proportion positive score.

By observation of TVS, then BR = .16, PPP = .814, NPP = .923

slide130

From a sample, observe

rate of positive scores.

Use TVS to estimate

condition BR in that

sample, PPP and NPP

for that BR.

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077

slide131

We found 726 defendants who completed BOTH RMT and WRT.

81/726 failed the RMT= .111 proportion positive score.

By observation of TVS, then BR = .16, PPP = .814, NPP = .923

If PPP = .814, then in this sample, the probability of feigning if RMT

is positive, is .814.

If NPP = .923, then in this sample, the probability of feigning if RMT

is negative is .077,

or 1 - .923.

  • To conduct MGV, we sampled from two groups:
  • The 645 individuals who passed the RMT—had a negative score.
  • The 81 individuals who failed the RMT—had a positive score.
slide132

Example of sampling

645 individuals with

negative scores, p(mal) = .077

81 individuals with

positive scores, p(mal) = .814

Sample n = 360

Sample n = 40

400 cases, 10% failures, 90% passes

Overall p(mal) = 40*.814 + 360*.077 = .151

Sample 25 times, plot x = .151, y = observed rate of

positive WRT scores, n for WLS = 400

slide133

For each sample, BR was pre-estimated. Then we observed

rate of positive WRT scores at each potential cut score.

slide134

Word Recognition Test (WRT)

Range 4 to 30, Mean = 23.2

Within group of RMT < 9, mean = 18.7

Within group of RMT > 8, mean = 23.8

slide135

Word Recognition Test (WRT)

For every potential cut score of WRT (4 -30), we plotted all x, y pairs

obtained from sampling

We performed WLS to obtain the FPR and TPR estimates

at every potential cut score.

We plotted the FPR and TPR estimates at every potential cut

score to generate the ROC curve.

AUC = .905, SE = .012, 95% CI for AUC = .881-.930.

Best cut scores:

LTE 18 (TPR = .563, FPR = .034)

LTE 19 (TPR = .620, FPR = .066)

slide137

We plotted the FPR and TPR estimates at every potential cut

score to generate the ROC curve.

AUC = .905, SE = .012, 95% CI for AUC = .881-.930.

Best cut scores:

LTE 18

(TPR = .563, FPR = .034)

LTE 19

(TPR = .620, FPR = .066)

TPR

FPR

WORD RECOGNITON TEST (WRT)

slide138

Summary:

  • We can use tests to form mixed groups for validation.
  • The best estimates of FPR and TPR for a test cut score
  • allow us to estimate PPP and NPP at our sample BR.
  • Instead of “known groups” design (which is misleading),
  • we do not presume to know (or care) about the status of
  • any individual. We assign individuals “probabilities of
  • having the condition” based on their test score.
  • Mixed groups have an overall “probability of having the
  • condition,” which is the average of the individual probabilities.
  • We do not need to be certain about group memberships.
  • We gain much flexibility by working with probabilities of having
  • the condition vs. certainties of having the condition.
slide140

Dawes 1967 showed that valid probability judgments are

excellent base rate indicators. His work was substantiated in Frederick 2000 and Frederick and Bowden 2009.

To conduct MGV, we formed groups of defendants for whom individuals ratings of likelihood of malingering psychosis were generated by forensic psychologists, before any testing took place.

The BR of malingered psychosis for each group was then the mean of the probability rating. If each member of the group had been rated as 10% likely to feign psychosis, then the BR of the group was estimated to be 10%.

slide141

We then observed the hit rate (proportion positive scores) for

the groups for a variety of F-family indicators of feigning on the

MMPI-2 and MMPI-2-RF.

We formed 15 groups of 30 individuals. For each group, we

had a static base rate, which was the mean of the probability

judgments assigned before testing.

Within each group, we iteratively observed the hit rate of positive

F-family indicators at each potential cut score. Using the BR

estimate and the proportion positive scores at each potential cut

score, we performed WLS to generate estimates of FPR and TPR.

From these estimates, we generated ROC curves.

slide142

15 groups, 30 defendants in each group, 450 defendants

Each defendant rated from 0 to 100 before testing, with respect

to likelihood he would feign psychosis.

Groups were formed after first sorting individuals by ratings, from

lowest to highest.

Mean ratings of groups (each group, n = 30):

0 0 1.2 4.2 5.0

5.0 5.0 5.0 8.1 10

14.5 22.2 30.3 45.7 72.3

Rates of positive F-family scores at each potential cut observed.

slide143

Estimates by Nicholson, Mouton, Bagby, Buis, Peterson,

and Buigas (1998):

AUC’s and SE: F (.929, .021) Fp (.885, .027)

slide145

Summary:

  • Using the estimates of likelihood of feigning based only
  • on clinician judgment prior to testing did not result in
  • random results. We can assume that mean probability
  • judgments were effective base rate estimates.
  • Our estimates of F and Fp are consistent with estimates
  • in large, well-validated analysis.
  • In this study, MMPI-2-RF indicators have higher mean
  • AUC and lower SE than their MMPI-2 counterparts.
slide147

f

Frequency Percent Valid Percent Cumulative Percent

Valid 4 4 2.7 3.1 3.1

6 1 .7 .8 3.8

9 1 .7 .8 4.6

10 2 1.3 1.5 6.1

11 3 2.0 2.3 8.4

12 2 1.3 1.5 9.9

13 3 2.0 2.3 12.2

14 4 2.7 3.1 15.3

15 2 1.3 1.5 16.8

16 3 2.0 2.3 19.1

17 3 2.0 2.3 21.4

18 2 1.3 1.5 22.9

19 4 2.7 3.1 26.0

20 5 3.4 3.8 29.8

21 2 1.3 1.5 31.3

22 7 4.7 5.3 36.6

23 3 2.0 2.3 38.9

24 1 .7 .8 39.7

25 5 3.4 3.8 43.5

26 6 4.0 4.6 48.1

27 6 4.0 4.6 52.7

28 7 4.7 5.3 58.0

29 2 1.3 1.5 59.5

30 2 1.3 1.5 61.1

31 6 4.0 4.6 65.6

32 5 3.4 3.8 69.5

33 3 2.0 2.3 71.8

34 4 2.7 3.1 74.8

35 4 2.7 3.1 77.9

36 3 2.0 2.3 80.2

37 7 4.7 5.3 85.5

38 4 2.7 3.1 88.5

39 2 1.3 1.5 90.1

40 3 2.0 2.3 92.4

41 3 2.0 2.3 94.7

43 1 .7 .8 95.4

45 1 .7 .8 96.2

46 1 .7 .8 96.9

47 1 .7 .8 97.7

48 2 1.3 1.5 99.2

52 1 .7 .8 100.0

Total 131 87.9 100.0

Missing System 18 12.1

Total 149 100.0

131 defendants

who took MMPI

and SIRS

slide148

52.7% of cases are 27 or lower

47.3% of cases are 28 or higher

What is the base rate of feigned psychopathology?

slide150

BR TPR FPR NPP PPP

What we say:

Within our sample of 131 defendants, the BR of feigned

psychopathology is .73 (NOT .475)

At BR = .73, the PPP of F GTE 28 is .976.

At BR = .73, the NPP of GTE 28 is .492, so

p(feigning if LTE 27) is still .508) (Remember, they’re

being given the SIRS for a reason)

slide151

F < 28

NPP about .66

slide154

Greve, Bianchini, Love, Brennan, & Heinly (2006) articulated six separate

groups with increasing base rate of malingering based on formal criteria

for malingering (the Slick criteria) to validate the MMPI-2 Fake Bad Scale

  • No incentive (no evidence of external incentive and no test
  • performance suggestive of malingering; n = 18, mean FBS = 15.4)
  • Incentive (external incentive, but no test performance suggestive
  • of malingering; n = 79, mean FBS = 19.5)
  • Suspect (external incentive and at least one indicator suggestive
  • of malingering; n = 66, mean FBS = 22.7)
  • Statistically Likely (external incentive; at least two indicators
  • suggestive of malingering; n = 51, mean FBS = 22.8)
  • Probable (external incentive; strong indicators of malingering;
  • n = 31, mean FBS = 26.9)
  • Definite (external incentive; very strong indicators of
  • malingering; n = 14, mean FBS = 29.8)
slide155

Even though it is clear that

  • BR Definite > BR Probable > BR Statistically Likely >
  • BR Suspect > BR Incentive Only > BR No Incentive
  • They were required, to conduct “Known” groups
  • validation, to ignore this obvious circumstance and to define
  • BR No Incentive = BR Incentive Only = 0
  • BR Statistically Likely = BR Probable = BR Definite = 1.0
  • And drop all participants defined as Suspect
  • to yield the following ROC
slide156

FBS ROC

generated

by “Known”

groups

validation by

Greve & Bianchini

slide157

If we had estimates of the BR for each of the subgroups

formed by Greve and Bianchini, we could use MGV to

estimate FPR and TPR for each potential cut score.

slide159

TOMM

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093

slide160

We can get estimates of BRs for those groups

from other work by Greve & Bianchini.

They formed similar groups using the Slick

criteria to investigate the TOMM.

We can use the proportion of positive TOMMs

in each of these subgroups to estimate the BRs

in each of them.

slide163

We take these BR estimates and reapply them

to the Greve & Bianchini FBS data.

slide164

Example of MGV for FBS based on BR estimates for

Greve & Bianchini groups established by Slick criteria

At FBS > 27

For FBS > 27, using WLS Regression, FPR = .091, TPR = .773

(For WLS, n is the weighted variable)

slide167

10 clinical

studies

using Rey

15-Item Test

No simulators

All clinical data

slide168

RMT validating

using MGV

with clinical

probability

judgments.

FPR = .025

TPR = .574

Frederick & Bowden,

2009

slide170

CI, TPR = .574,

SE = .044

We will

generate TVS

based on these

values and

find PPP and

1 – NPP to

estimate

probability of

bad intent

represented

by RMT score.

vip verbal subtest items
VIP Verbal Subtest Items

Easy: Baby Drink Infant

Moderate: People Ally Folk

Difficult: Nimiety Conceit Surfeit

slide176

Not guessing,

knowledgeable

responding

Guessing

Guessing

is imminent

Easy items

Difficult items

slide178

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077

slide179

Here we

are matching

VIP categories

to the construct

most likely

captured by

the VIP.

Points in

scatterplot

represent

groups of 25

individuals.

Sorted defendants

by clinical ratings

of malingering,

then took 20

groups of 25 and

one group of 27,

for 527 defendants.

BR of .42 estimated

for this group is mean

of PPP for positive RMT

scores in this group and

(1 – NPP) for negative

RMT scores in the group

slide180

Same 21

subgroups,

N = 527

defendants

slide181

527 criminal

defendants

VRIN was converted

to “probability

of invalid

responding” by

dividing VRIN

raw score by 12.

VRIN raw scores

>12 were

assigned p = 1.

We are interested

in FPR and TPR for

“Invalid”