Solving classification problems for symptom validity tests with mixed groups validation
Download
1 / 181

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation - PowerPoint PPT Presentation


  • 285 Views
  • Uploaded on

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation. Richard Frederick, Ph.D., ABPP (Forensic) US Medical Center for Federal Prisoners Springfield, Missouri. I am not a neuropsychologist. My view of brain. Your view of brain. My board certifications:.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation' - mohammad-gamble


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Solving classification problems for symptom validity tests with mixed groups validation

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation

Richard Frederick, Ph.D., ABPP (Forensic)

US Medical Center for Federal Prisoners

Springfield, Missouri


I am not a neuropsychologist with .

My view of brain

Your view of brain


My board certifications

My board certifications: with

Forensic Psychology

American Board of Professional Psychology

Assessment Psychology

American Board of Assessment Psychology


My professional goal

My professional goal: with

Use tests properly in forensic psychological assessments


Goals of workshop with

Participants in this workshop will be able

to employ Excel graphing methods:

--to evaluate classification characteristics of

symptom validity tests

--to adapt symptom validity test scores to their

individual, local, base rates

--to combine information from local base rate

and multiple symptom validity tests



Something is terribly wrong
Something is terribly wrong with

  • The SIRS has sensitivity = .485 and specificity = .995.

  • The SIRS was administered to 131 criminal defendants

  • who were strongly suspected of feigned psychopathology.

  • 68% of them were categorized as feigning by the SIRS


What is a classification test? with

A structured routine for determining

which individuals belong to which

of two groups.


  • There are two groups. with

  • (2) It’s not easy to determine which

  • group an individual belongs to

  • without the help of the test.



The distributions represent our with

estimations of how the populations of

the two groups score on the test.

We generally estimate the population

distributions by sampling. We notice

that the populations have two separate,

but overlapping distributions. The extent

of the overlap is of concern to us.


  • Questions that must be addressed in with

  • research before we can continue:

  • Are there really two separate groups?

  • Can we effectively represent the

  • population distributions by sampling?



What we notice next. with

The mean separation between the

groups is 10 points.

Persons in Population A have a mean

score that is 10 points below persons in

Population B.

The sd for each population is the same. The

mean separation between groups is one sd.


When researchers talk about mean with

separation, they often refer to effect size.

Often, Cohen’s d is the statistic used to

refer to standardized mean separation.

Here, Cohen’s d = 1. This is often referred

to as a large, or very large, effect size.


Mean separation = 0 with

Making tests often means finding those characteristics that best separate the distributions of the two groups.

Two distributions of gender

with respect to:

Intelligence


Moderately large mean separation with

Two distributions of gender

with respect to:

Longevity


Large mean separation with

Two distributions of gender

with respect to:

Hair Length


Very large mean separation with

Two distributions of gender

with respect to:

Body Mass



  • Summary: with

  • We have two groups.

  • We have a test for which the two

  • groups score differentially.

  • (3) The differences in mean scores

  • represents a very large effect.



More commonly, researchers report with

Sensitivity and Specificity. These terms

are common, but not most helpful.

We are going to use the terms:

True Positive Rate (TPR) and

False Positive Rate (FPR).

TPR = Sensitivity

FPR = 1 - Specificity


What are TPR and FPR? with

TPR is the proportion of individuals who do have the condition who generate positive scores. TPR is the rate of scores are beyond the cut in the direction that indicates the presence of the condition.

FPR is the proportion of individuals who do NOT have the condition who generate positive scores. FPR is the rate of scores beyond the cut in the direction that indicates the presence of the condition.


The green line represents with

the cut score. Scores to the

LEFT of the line are classified

NEGATIVE. Scores to right

are classified POSITIVE.

Have nots

Haves

Here, the False Positive Rate is 92.4%.

The True Positive Rate is 100%.

As we move the line to the right, both

rates DECREASE.


To totally eliminate with

false positives, we have

to be willing to identify

almost no one as a

positive.


TPR = True Positives/Haves with

FPR = False Positives/Have Nots


Haves with

Have nots


A positive score will be one that is with

associated with Population A

membership. If we set a point at which

a score will be used to say, “This score

represents Population A,” such a score

will be referred to as a “positive score.”

A positive score can be a true positive

or a false positive: unknown to us.


The True Positive Rate is the with proportion of

Population A members who generate a

positive score.

In our figure, the point at which we

begin to identify “positive scores” is at 50, the

mean of population A. Scores at or below 50

are called positive, and a person who

generates a positive score is classified as a

Population A member.


We can pick any value to be our “cut score,” but with

it’s hard to pick one that doesn’t result in some

Population B members producing “positive

scores.”

In our figure, 50% of the Population A members

have scores at 50 or below. This is the True

Positive Rate. TPR = .50.

In our figure, 16% of the Population B members

have scores at 50 or below. This it the False

Positive Rate. FPR = .16.


We note that it is with not the test that has

a certain TPR and FPR.

It is the chosen test score that has a

certain TPR and FPR.

A different test score will almost

certainly have different TPR and FPR.


Overcoming limiting factors of “known groups” validation in determining test score sensitivity and specificity


We think of a test as a way to characterize a dependency. in determining test score sensitivity and specificity

As you have more of X, you have more of Y.

Y depends on X.

X predicts Y.

X is some construct. Y is some test score.

There is a relationship that we wish to characterize and

quantify.


Let’s consider feigning. in determining test score sensitivity and specificity

As you are more likely to feign, you are more likely to

engage in certain behavior.

This behavior might be “providing answers to items on a

test” at a certain rate.

You might choose more items, you might choose fewer

items than “normals.”


We develop the idea that we can identify individuals who in determining test score sensitivity and specificity

respond at a certain rate as feigners, and we decide to

make a decision point about when we call test takers

feigners and when we don’t.

We call that decision point a cut score.

We call test scores at or beyond the cut score:

positive scores

Some positive scores are correct: true positives

Some positive scores are incorrect: false positives


If our test is any good, and if the relationship between in determining test score sensitivity and specificity

X and Y is strong, then our rate of true positives is much

higher than our rate of false positives.

Let’s skip to the end. We are now using the test in our

clinic.

We look over our results. We see a number of “positive

scores.”

We know that those “positive scores” are some unknown

mixture of “true positives” and “false positives.”

We’d like to know what that ratio of that mixture is.


Here’s how we do it: in determining test score sensitivity and specificity

First, we estimate what the true positive rate of the cut

score is.

Then, we estimate what the false positive rate of the

cut score is.

Then, we figure out what percentage of people in our

sample are feigning.

Then we can get the ratio of the mixture of our true

positive and false positives in all the positive scores in our

clinic. (We call this positive predictive power.)


Getting TPR and FPR: in determining test score sensitivity and specificity

We depend on researchers to tell us what the estimates

of true positive rate and false positive rate are.

They usually do this through a process called

“criterion groups validation.”

People with more confidence than might be called

for refer to this process as “known groups validation.”


The process is seemingly straightforward. in determining test score sensitivity and specificity

Identify two groups. One group has the condition.

All the positives in this group are “true positives.”

One group doesn’t have the condition. All the positives

in this group are “false positives.”

The rate of “true positives” is the sensitivity of the test.

TPR = sensitivity.

The rate of “false positives” is the non-specificity of

the test. FPR = 1 – specificity.


There are many problems with this process, but let’s in determining test score sensitivity and specificity

focus on the main two.

Problem 1

In Study 1, for a given cut score, researchers report the

TPR is .67 and the FPR is .12.

In Study 2, for the same cut score, researchers report

TPR = .58 and FPR = .09.

Which values do you use?


Problem 2: in determining test score sensitivity and specificity

In Study 1, for a given cut score, researchers report the

TPR is .67 and the FPR is .12.

In Study 2, for a different cut score, researchers report

TPR = .58 and FPR = .09.

Which cut score do you use?


Known groups validation
“Known” groups validation in determining test score sensitivity and specificity


Let s validate a test
Let’s validate a test! in determining test score sensitivity and specificity



We take our best shot at a test
We take our best shot at a test. responders and 100 feigners.

TRUTH

TEST


Test results
Test results responders and 100 feigners.

TRUTH

TEST


We say for our test true positive rate 49 100 49 sens 49 false positive rate 1 100 1 specificity 99
We say for our test: responders and 100 feigners.True positive rate = 49/100 = 49% [sens = 49%]False positive rate = 1/100 = 1% [specificity = 99%]

TRUTH

TEST


Because God does not whisper to us anymore, responders and 100 feigners.we take this test, our best test, and we say, “This is the best we can do.” Let’s call it our Gold Standard.We will now make criterion groups with this test,and we will call the groups “Known Groups.”We will then validate tests, based on these Known Groups.


We say for our test: responders and 100 feigners.True positive rate = 49/100 = 49% [sensitivity = 49%]False positive rate = 1/100 = 1% [specificity = 99%]

TRUTH

TEST


Our move from truth to known groups
Our move from TRUTH to KNOWN GROUPS responders and 100 feigners.

“KNOWN” GROUPS

TRUTH



Let’s validate a new test, which standardjust happens to be a perfect test. What test diagnostic efficiencies will we assign our new, perfect, test?

“KNOWN” GROUPS

PERFECT TEST


Let’s validate a new test, which standardjust happens to be a perfect test. What test diagnostic efficiencies will we assign our new, perfect, test?

“KNOWN” GROUPS

Our belief that

we can make

perfect criterion

groups from

imperfect criteria

has led us to

misunderstand

tremendously

what we are

doing.

PERFECT TEST

TPR = 49/50 = 98%, FPR = 51/150 = 34%


Let’s begin to address these problems in a standard

non-traditional way.








REMINDER: Here is what we are working on— standardfiguring out

which positives in our clinic are true positives.

First, we estimate what the true positive rate of the cut

score is. Then, we estimate what the false positive rate of

the cut score is.

Let’s do that part now.

Then, we figure out what percentage of people in our

sample are feigning.

Then we can get the ratio of the mixture of our true

positive and false positives in all the positive scores in our

clinic.





When BR = 0, 10% of scores standard

positive, all false positives


When we standard

say FPR = .16

and TPR = .50,

what we’re

saying is that,

no matter

what samples

we test, we

expect to

see no fewer

than 16%

positive scores

and no more

than 50%

positive

scores.

Movement along this line from left to right

represents increasing rate of Population A and

increasing rate of positive scores.


FPR = .10 standard

TPR = .80



TOMM standard

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093


For any standard

imperfect

test,

PPP ranges

from 0 to 1

as base rate

ranges from

0 to 1

NPP ranges

from 0 to 1

as base rate

ranges from

1 to 0

NPP

PPP


Using mgv to estimate test diagnostic efficiencies of the reliable digit span

Using MGV to estimate standardtest diagnostic efficiencies of the Reliable Digit Span

Laurie Ragatz, PhD

Richard Frederick, PhD


What is reliable digit span
What is Reliable Digit Span? standard

RDS is a symptom validity measure for Digit Span. The value of RDS is derived by adding longest strings of two trials passed for both

forward and backward Digit Span.

Researched cut scores include 5 or lower, 6 or lower, 7 or lower, or 8 or lower.


Reliable digit span example
Reliable Digit Span Example: standard

Directions: Examinee recalls numbers in the same order they were provided by the examiner

Directions: Examinee recalls numbers in the reverse order they were provided by the examiner

Reliable Digit Span: 4 + 3 =7


(1) We found all available articles dealing standard

with RDS and identified the cut scores

investigated. We included simulator studies.

(2) Based on the authors’ decision about

criterion group membership, we

calculated the overall base rate of

malingering in the study.

(3) We observed the overall rate of

positive scores in the study at the

identified cut score.


(4) We did not include any data for persons standard

with mental retardation. The rate of

positive scores among persons with mental

retardation was exceedingly high for all cut

scores.


Criterion group standard

Test outcome

We have 63 malingerers in a sample of 203. BR = 63/203 = 0.31.

We have 57 positive scores. Proportion positive scores (PPS) is 57/203 = .28. For this study, we plot (BR, PPS) = (.31, .28)

x = .31, y = .28. Our n for WLS = 203.


Rds 5 or lower
RDS = 5 or lower standard

Using weighted least squares regression (with N as the weight),

we regressed Proportion Positive Scores (PPS) on Base Rate (BR)

to generate the Proportion Positive Score Line.

We obtained y-intercept of -.015 (all negative values are

truncated to 0), and slope of .265.


RDS = 5 or lower standard

put these data in WLS

to obtain regression

line characteristics

scatterplot



RDS = 6 or lower standard

y-intercept = .015, slope = .419



RDS = 7 or lower standard

y-intercept = .187, slope = .39



RDS = 8 or lower standard

y-intercept = .236, slope = .824



As we move standard

from a cut score

of 5 or lower to 6

or lower, we

obtain

substantial

improvement in

TPR estimate

with little cost

in FPR increase.


Our choice for best cut score for RDS standard

RDS: 6 or lower, FPR = .015, TPR = .434


By using WLS regression, we can obtain standard errors of standard

our estimates of FPR and TPR.

So, new researchers can test hypotheses about parametric

values of FPR and TPR.


Overcoming limiting factors of “known groups” validation in determining test score sensitivity and specificity


  • Summary: in determining test score sensitivity and specificity

  • The TVS and MGV allow powerful research into existing published data sets. Summary data are used.

  • Understanding of parametric values of TPR and FPR is facilitated when researchers publish results on a variety of cut scores that should be considered. A frequency

  • distribution would be ideal, for example,


  • Combining studies in this way allows us to generate in determining test score sensitivity and specificity

  • stable values of TPR and FPR with SE’s so that new research

  • can test those values.

  • Researchers should focus on the basis for estimating BR’s

  • in their research groups. All research estimating FPR and TPR

  • is vulnerable to error when the purity of research groups is

  • overestimated. Working towards a reliable estimate of

  • mixed group base rate will facilitate better validation studies.


Reliably estimate local base rates of feigning for proper allocation of sensitivity and specificity information


  • How can the Test Validation Summary help me allocation of sensitivity and specificity information

  • determine my local BR?

  • Get the best estimate of the test FPR and TPR

  • for a certain test score.

  • 2. Find the proportion of test scores in your

  • sample that are positive scores.


From a sample, observe allocation of sensitivity and specificity information

rate of positive scores.

Use TVS to estimate

condition BR in that

sample, PPP and NPP

for that BR.

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077


TOMM allocation of sensitivity and specificity information

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093


Beth A. Caillouet, Bernice A. Marcopulos, Jesse G. Brand, allocation of sensitivity and specificity information

Julie Ann Kent, & Richard I. Frederick

Question: What are the BRs of malingering in the two samples?


Question: What are the BRs of malingering in the two samples?

Information needed:

Estimates of TOMM FPR and TPR. From TOMM TVS, we

get FPR = .056, TPR = 742.

Sample 1: Secondary gain present. Proportion positive

scores = 55/220 = .25.

Sample 2: Secondary gain absent. Proportion positive

scores = 34/299 = .11.

Use TOMM TVS to estimate BR of each sample.


When PPS = .25, samples?

BR = .28.

When PPS = .11,

BR = .08.


Defensibly choose symptom validity cut scores that are ideally suited for their local base rates
Defensibly choose symptom validity cut scores that are ideally suited for their local base rates


M fast
M-FAST ideally suited for their local base rates


TPR = .93 FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


TPR = .93 FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


TPR = .93 FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


TPR = .93 FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


TPR = .93 FPR = 10/56 = .18 ideally suited for their local base rates

BR malingering = 35%, N = 86


TPR = 28/30 = .93 FPR = 10/56 = .18 ideally suited for their local base rates

PPP = 28/38 = .737 NPP = .958

BR malingering = .35

NPP PPP 1 – FPR TPR


TPR = 28/30 = .93 FPR = 10/56 = .18 ideally suited for their local base rates

PPP = 28/131 = .213 NPP = .996

BR malingering = .05

NPP PPP 1 – FPR TPR


Test validation ideally suited for their local base rates

summary for

M-FAST cut

score

recommended

by test manual.

PPP does not even reach 50%

correct decisions until BR > .16

M-FAST > 5

FPR = .17

TPR = .93

At recommended cut score FPR very high


At BR = .05, PPP does not exceed ideally suited for their local base rates

.50 until cut score adjusted to

> 9 on M-FAST


Combining information from local base ideally suited for their local base rates

rate and multiple symptom validity tests

You can get estimates of PPP and NPP

for the sample you work with—IF you

can reliably estimate the BR.


737 defendants were administered: ideally suited for their local base rates

Rey 15 Item Memory Test (RMT)—memorize and

reproduce 15 items—very easy test.

Score is items reproduced (0 to 15)

Word Recognition Test (WRT)—memorize 15 words,

identify those 15 and correctly reject 15 from a list of 30.

Score is number of hits and correct rejections (0 to 30)


RMT validating ideally suited for their local base rates

using MGV

with clinical

probability

judgments.

FPR = .025

TPR = .574

Frederick & Bowden,

2009


RMT < 9 ideally suited for their local base rates

FPR = .025

TPR = .574

We found 726 defendants who completed BOTH RMT and WRT.

81/726 failed the RMT= .111 proportion positive score.

By observation of TVS, then BR = .16, PPP = .814, NPP = .923


From a sample, observe ideally suited for their local base rates

rate of positive scores.

Use TVS to estimate

condition BR in that

sample, PPP and NPP

for that BR.

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077


We found 726 defendants who completed BOTH RMT and WRT. ideally suited for their local base rates

81/726 failed the RMT= .111 proportion positive score.

By observation of TVS, then BR = .16, PPP = .814, NPP = .923

If PPP = .814, then in this sample, the probability of feigning if RMT

is positive, is .814.

If NPP = .923, then in this sample, the probability of feigning if RMT

is negative is .077,

or 1 - .923.

  • To conduct MGV, we sampled from two groups:

  • The 645 individuals who passed the RMT—had a negative score.

  • The 81 individuals who failed the RMT—had a positive score.


Example of sampling ideally suited for their local base rates

645 individuals with

negative scores, p(mal) = .077

81 individuals with

positive scores, p(mal) = .814

Sample n = 360

Sample n = 40

400 cases, 10% failures, 90% passes

Overall p(mal) = 40*.814 + 360*.077 = .151

Sample 25 times, plot x = .151, y = observed rate of

positive WRT scores, n for WLS = 400


For each sample, BR was pre-estimated. Then we observed ideally suited for their local base rates

rate of positive WRT scores at each potential cut score.


Word Recognition Test (WRT) ideally suited for their local base rates

Range 4 to 30, Mean = 23.2

Within group of RMT < 9, mean = 18.7

Within group of RMT > 8, mean = 23.8


Word Recognition Test (WRT) ideally suited for their local base rates

For every potential cut score of WRT (4 -30), we plotted all x, y pairs

obtained from sampling

We performed WLS to obtain the FPR and TPR estimates

at every potential cut score.

We plotted the FPR and TPR estimates at every potential cut

score to generate the ROC curve.

AUC = .905, SE = .012, 95% CI for AUC = .881-.930.

Best cut scores:

LTE 18 (TPR = .563, FPR = .034)

LTE 19 (TPR = .620, FPR = .066)


We plotted the FPR and TPR estimates at every potential cut ideally suited for their local base rates

score to generate the ROC curve.

AUC = .905, SE = .012, 95% CI for AUC = .881-.930.

Best cut scores:

LTE 18

(TPR = .563, FPR = .034)

LTE 19

(TPR = .620, FPR = .066)

TPR

FPR

WORD RECOGNITON TEST (WRT)


  • Summary: ideally suited for their local base rates

  • We can use tests to form mixed groups for validation.

  • The best estimates of FPR and TPR for a test cut score

  • allow us to estimate PPP and NPP at our sample BR.

  • Instead of “known groups” design (which is misleading),

  • we do not presume to know (or care) about the status of

  • any individual. We assign individuals “probabilities of

  • having the condition” based on their test score.

  • Mixed groups have an overall “probability of having the

  • condition,” which is the average of the individual probabilities.

  • We do not need to be certain about group memberships.

  • We gain much flexibility by working with probabilities of having

  • the condition vs. certainties of having the condition.


Another example
Another example ideally suited for their local base rates


Dawes 1967 showed that valid probability judgments are ideally suited for their local base rates

excellent base rate indicators. His work was substantiated in Frederick 2000 and Frederick and Bowden 2009.

To conduct MGV, we formed groups of defendants for whom individuals ratings of likelihood of malingering psychosis were generated by forensic psychologists, before any testing took place.

The BR of malingered psychosis for each group was then the mean of the probability rating. If each member of the group had been rated as 10% likely to feign psychosis, then the BR of the group was estimated to be 10%.


We then observed the hit rate (proportion positive scores) for

the groups for a variety of F-family indicators of feigning on the

MMPI-2 and MMPI-2-RF.

We formed 15 groups of 30 individuals. For each group, we

had a static base rate, which was the mean of the probability

judgments assigned before testing.

Within each group, we iteratively observed the hit rate of positive

F-family indicators at each potential cut score. Using the BR

estimate and the proportion positive scores at each potential cut

score, we performed WLS to generate estimates of FPR and TPR.

From these estimates, we generated ROC curves.


15 groups, 30 defendants in each group, 450 defendants for

Each defendant rated from 0 to 100 before testing, with respect

to likelihood he would feign psychosis.

Groups were formed after first sorting individuals by ratings, from

lowest to highest.

Mean ratings of groups (each group, n = 30):

0 0 1.2 4.2 5.0

5.0 5.0 5.0 8.1 10

14.5 22.2 30.3 45.7 72.3

Rates of positive F-family scores at each potential cut observed.


Estimates by Nicholson, Mouton, Bagby, Buis, Peterson, for

and Buigas (1998):

AUC’s and SE: F (.929, .021) Fp (.885, .027)


  • Summary: for

  • Using the estimates of likelihood of feigning based only

  • on clinician judgment prior to testing did not result in

  • random results. We can assume that mean probability

  • judgments were effective base rate estimates.

  • Our estimates of F and Fp are consistent with estimates

  • in large, well-validated analysis.

  • In this study, MMPI-2-RF indicators have higher mean

  • AUC and lower SE than their MMPI-2 counterparts.



f for

Frequency Percent Valid Percent Cumulative Percent

Valid 4 4 2.7 3.1 3.1

6 1 .7 .8 3.8

9 1 .7 .8 4.6

10 2 1.3 1.5 6.1

11 3 2.0 2.3 8.4

12 2 1.3 1.5 9.9

13 3 2.0 2.3 12.2

14 4 2.7 3.1 15.3

15 2 1.3 1.5 16.8

16 3 2.0 2.3 19.1

17 3 2.0 2.3 21.4

18 2 1.3 1.5 22.9

19 4 2.7 3.1 26.0

20 5 3.4 3.8 29.8

21 2 1.3 1.5 31.3

22 7 4.7 5.3 36.6

23 3 2.0 2.3 38.9

24 1 .7 .8 39.7

25 5 3.4 3.8 43.5

26 6 4.0 4.6 48.1

27 6 4.0 4.6 52.7

28 7 4.7 5.3 58.0

29 2 1.3 1.5 59.5

30 2 1.3 1.5 61.1

31 6 4.0 4.6 65.6

32 5 3.4 3.8 69.5

33 3 2.0 2.3 71.8

34 4 2.7 3.1 74.8

35 4 2.7 3.1 77.9

36 3 2.0 2.3 80.2

37 7 4.7 5.3 85.5

38 4 2.7 3.1 88.5

39 2 1.3 1.5 90.1

40 3 2.0 2.3 92.4

41 3 2.0 2.3 94.7

43 1 .7 .8 95.4

45 1 .7 .8 96.2

46 1 .7 .8 96.9

47 1 .7 .8 97.7

48 2 1.3 1.5 99.2

52 1 .7 .8 100.0

Total 131 87.9 100.0

Missing System 18 12.1

Total 149 100.0

131 defendants

who took MMPI

and SIRS


52.7% of cases are 27 or lower for

47.3% of cases are 28 or higher

What is the base rate of feigned psychopathology?


BR TPR FPR NPP PPP for

What we say:

Within our sample of 131 defendants, the BR of feigned

psychopathology is .73 (NOT .475)

At BR = .73, the PPP of F GTE 28 is .976.

At BR = .73, the NPP of GTE 28 is .492, so

p(feigning if LTE 27) is still .508) (Remember, they’re

being given the SIRS for a reason)


F < 28 for

NPP about .66


F > 27 for



Greve, Bianchini, Love, Brennan, & for Heinly (2006) articulated six separate

groups with increasing base rate of malingering based on formal criteria

for malingering (the Slick criteria) to validate the MMPI-2 Fake Bad Scale

  • No incentive (no evidence of external incentive and no test

  • performance suggestive of malingering; n = 18, mean FBS = 15.4)

  • Incentive (external incentive, but no test performance suggestive

  • of malingering; n = 79, mean FBS = 19.5)

  • Suspect (external incentive and at least one indicator suggestive

  • of malingering; n = 66, mean FBS = 22.7)

  • Statistically Likely (external incentive; at least two indicators

  • suggestive of malingering; n = 51, mean FBS = 22.8)

  • Probable (external incentive; strong indicators of malingering;

  • n = 31, mean FBS = 26.9)

  • Definite (external incentive; very strong indicators of

  • malingering; n = 14, mean FBS = 29.8)


  • Even though it is clear that for

  • BR Definite > BR Probable > BR Statistically Likely >

  • BR Suspect > BR Incentive Only > BR No Incentive

  • They were required, to conduct “Known” groups

  • validation, to ignore this obvious circumstance and to define

  • BR No Incentive = BR Incentive Only = 0

  • BR Statistically Likely = BR Probable = BR Definite = 1.0

  • And drop all participants defined as Suspect

  • to yield the following ROC


FBS ROC for

generated

by “Known”

groups

validation by

Greve & Bianchini


If we had estimates of the BR for each of the subgroups for

formed by Greve and Bianchini, we could use MGV to

estimate FPR and TPR for each potential cut score.


We have our stable estimate of for

TOMM FPR and TPR


TOMM for

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093


We can get estimates of BRs for those groups for

from other work by Greve & Bianchini.

They formed similar groups using the Slick

criteria to investigate the TOMM.

We can use the proportion of positive TOMMs

in each of these subgroups to estimate the BRs

in each of them.



We take these BR estimates and reapply them for

to the Greve & Bianchini FBS data.


Example of MGV for FBS based on BR estimates for for

Greve & Bianchini groups established by Slick criteria

At FBS > 27

For FBS > 27, using WLS Regression, FPR = .091, TPR = .773

(For WLS, n is the weighted variable)


Evaluate constructs that underlie symptom validity tests
Evaluate constructs that underlie for symptom validity tests


10 clinical for

studies

using Rey

15-Item Test

No simulators

All clinical data


RMT validating for

using MGV

with clinical

probability

judgments.

FPR = .025

TPR = .574

Frederick & Bowden,

2009


CI, TPR = .574, for

SE = .044

We will

generate TVS

based on these

values and

find PPP and

1 – NPP to

estimate

probability of

bad intent

represented

by RMT score.



Vip verbal subtest items
VIP Verbal Subtest Items for

Easy: Baby Drink Infant

Moderate: People Ally Folk

Difficult: Nimiety Conceit Surfeit



Not guessing, for

knowledgeable

responding

Guessing

Guessing

is imminent

Easy items

Difficult items



527 criminal for

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077


Here we for

are matching

VIP categories

to the construct

most likely

captured by

the VIP.

Points in

scatterplot

represent

groups of 25

individuals.

Sorted defendants

by clinical ratings

of malingering,

then took 20

groups of 25 and

one group of 27,

for 527 defendants.

BR of .42 estimated

for this group is mean

of PPP for positive RMT

scores in this group and

(1 – NPP) for negative

RMT scores in the group


Same 21 for

subgroups,

N = 527

defendants


527 criminal for

defendants

VRIN was converted

to “probability

of invalid

responding” by

dividing VRIN

raw score by 12.

VRIN raw scores

>12 were

assigned p = 1.

We are interested

in FPR and TPR for

“Invalid”


ad