Solving classification problems for symptom validity tests with mixed groups validation
Download
1 / 181

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation - PowerPoint PPT Presentation


  • 274 Views
  • Uploaded on
  • Presentation posted in: General

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation. Richard Frederick, Ph.D., ABPP (Forensic) US Medical Center for Federal Prisoners Springfield, Missouri. I am not a neuropsychologist. My view of brain. Your view of brain. My board certifications:.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Solving classification problems for symptom validity tests with mixed groups validation
Solving Classification Problems for Symptom Validity Tests with Mixed Groups Validation

Richard Frederick, Ph.D., ABPP (Forensic)

US Medical Center for Federal Prisoners

Springfield, Missouri


Solving classification problems for symptom validity tests with mixed groups validation

I am not a neuropsychologist with .

My view of brain

Your view of brain


My board certifications
My board certifications: with

Forensic Psychology

American Board of Professional Psychology

Assessment Psychology

American Board of Assessment Psychology


My professional goal
My professional goal: with

Use tests properly in forensic psychological assessments


Solving classification problems for symptom validity tests with mixed groups validation

Goals of workshop with

Participants in this workshop will be able

to employ Excel graphing methods:

--to evaluate classification characteristics of

symptom validity tests

--to adapt symptom validity test scores to their

individual, local, base rates

--to combine information from local base rate

and multiple symptom validity tests



Something is terribly wrong
Something is terribly wrong with

  • The SIRS has sensitivity = .485 and specificity = .995.

  • The SIRS was administered to 131 criminal defendants

  • who were strongly suspected of feigned psychopathology.

  • 68% of them were categorized as feigning by the SIRS


Solving classification problems for symptom validity tests with mixed groups validation

What is a classification test? with

A structured routine for determining

which individuals belong to which

of two groups.


Solving classification problems for symptom validity tests with mixed groups validation

  • There are two groups. with

  • (2) It’s not easy to determine which

  • group an individual belongs to

  • without the help of the test.



Solving classification problems for symptom validity tests with mixed groups validation

The distributions represent our with

estimations of how the populations of

the two groups score on the test.

We generally estimate the population

distributions by sampling. We notice

that the populations have two separate,

but overlapping distributions. The extent

of the overlap is of concern to us.


Solving classification problems for symptom validity tests with mixed groups validation

  • Questions that must be addressed in with

  • research before we can continue:

  • Are there really two separate groups?

  • Can we effectively represent the

  • population distributions by sampling?



Solving classification problems for symptom validity tests with mixed groups validation

What we notice next. with

The mean separation between the

groups is 10 points.

Persons in Population A have a mean

score that is 10 points below persons in

Population B.

The sd for each population is the same. The

mean separation between groups is one sd.


Solving classification problems for symptom validity tests with mixed groups validation

When researchers talk about mean with

separation, they often refer to effect size.

Often, Cohen’s d is the statistic used to

refer to standardized mean separation.

Here, Cohen’s d = 1. This is often referred

to as a large, or very large, effect size.


Solving classification problems for symptom validity tests with mixed groups validation

Mean separation = 0 with

Making tests often means finding those characteristics that best separate the distributions of the two groups.

Two distributions of gender

with respect to:

Intelligence


Solving classification problems for symptom validity tests with mixed groups validation

Moderately large mean separation with

Two distributions of gender

with respect to:

Longevity


Solving classification problems for symptom validity tests with mixed groups validation

Large mean separation with

Two distributions of gender

with respect to:

Hair Length


Solving classification problems for symptom validity tests with mixed groups validation

Very large mean separation with

Two distributions of gender

with respect to:

Body Mass



Solving classification problems for symptom validity tests with mixed groups validation

  • Summary: with

  • We have two groups.

  • We have a test for which the two

  • groups score differentially.

  • (3) The differences in mean scores

  • represents a very large effect.



Solving classification problems for symptom validity tests with mixed groups validation

More commonly, researchers report with

Sensitivity and Specificity. These terms

are common, but not most helpful.

We are going to use the terms:

True Positive Rate (TPR) and

False Positive Rate (FPR).

TPR = Sensitivity

FPR = 1 - Specificity


Solving classification problems for symptom validity tests with mixed groups validation

What are TPR and FPR? with

TPR is the proportion of individuals who do have the condition who generate positive scores. TPR is the rate of scores are beyond the cut in the direction that indicates the presence of the condition.

FPR is the proportion of individuals who do NOT have the condition who generate positive scores. FPR is the rate of scores beyond the cut in the direction that indicates the presence of the condition.


Solving classification problems for symptom validity tests with mixed groups validation

The green line represents with

the cut score. Scores to the

LEFT of the line are classified

NEGATIVE. Scores to right

are classified POSITIVE.

Have nots

Haves

Here, the False Positive Rate is 92.4%.

The True Positive Rate is 100%.

As we move the line to the right, both

rates DECREASE.


Solving classification problems for symptom validity tests with mixed groups validation

To totally eliminate with

false positives, we have

to be willing to identify

almost no one as a

positive.


Solving classification problems for symptom validity tests with mixed groups validation

TPR = True Positives/Haves with

FPR = False Positives/Have Nots


Solving classification problems for symptom validity tests with mixed groups validation

Haves with

Have nots


Solving classification problems for symptom validity tests with mixed groups validation

A positive score will be one that is with

associated with Population A

membership. If we set a point at which

a score will be used to say, “This score

represents Population A,” such a score

will be referred to as a “positive score.”

A positive score can be a true positive

or a false positive: unknown to us.


Solving classification problems for symptom validity tests with mixed groups validation

The True Positive Rate is the with proportion of

Population A members who generate a

positive score.

In our figure, the point at which we

begin to identify “positive scores” is at 50, the

mean of population A. Scores at or below 50

are called positive, and a person who

generates a positive score is classified as a

Population A member.


Solving classification problems for symptom validity tests with mixed groups validation

We can pick any value to be our “cut score,” but with

it’s hard to pick one that doesn’t result in some

Population B members producing “positive

scores.”

In our figure, 50% of the Population A members

have scores at 50 or below. This is the True

Positive Rate. TPR = .50.

In our figure, 16% of the Population B members

have scores at 50 or below. This it the False

Positive Rate. FPR = .16.


Solving classification problems for symptom validity tests with mixed groups validation

We note that it is with not the test that has

a certain TPR and FPR.

It is the chosen test score that has a

certain TPR and FPR.

A different test score will almost

certainly have different TPR and FPR.


Solving classification problems for symptom validity tests with mixed groups validation
Overcoming limiting factors of “known groups” validation in determining test score sensitivity and specificity


Solving classification problems for symptom validity tests with mixed groups validation

We think of a test as a way to characterize a dependency. in determining test score sensitivity and specificity

As you have more of X, you have more of Y.

Y depends on X.

X predicts Y.

X is some construct. Y is some test score.

There is a relationship that we wish to characterize and

quantify.


Solving classification problems for symptom validity tests with mixed groups validation

Let’s consider feigning. in determining test score sensitivity and specificity

As you are more likely to feign, you are more likely to

engage in certain behavior.

This behavior might be “providing answers to items on a

test” at a certain rate.

You might choose more items, you might choose fewer

items than “normals.”


Solving classification problems for symptom validity tests with mixed groups validation

We develop the idea that we can identify individuals who in determining test score sensitivity and specificity

respond at a certain rate as feigners, and we decide to

make a decision point about when we call test takers

feigners and when we don’t.

We call that decision point a cut score.

We call test scores at or beyond the cut score:

positive scores

Some positive scores are correct: true positives

Some positive scores are incorrect: false positives


Solving classification problems for symptom validity tests with mixed groups validation

If our test is any good, and if the relationship between in determining test score sensitivity and specificity

X and Y is strong, then our rate of true positives is much

higher than our rate of false positives.

Let’s skip to the end. We are now using the test in our

clinic.

We look over our results. We see a number of “positive

scores.”

We know that those “positive scores” are some unknown

mixture of “true positives” and “false positives.”

We’d like to know what that ratio of that mixture is.


Solving classification problems for symptom validity tests with mixed groups validation

Here’s how we do it: in determining test score sensitivity and specificity

First, we estimate what the true positive rate of the cut

score is.

Then, we estimate what the false positive rate of the

cut score is.

Then, we figure out what percentage of people in our

sample are feigning.

Then we can get the ratio of the mixture of our true

positive and false positives in all the positive scores in our

clinic. (We call this positive predictive power.)


Solving classification problems for symptom validity tests with mixed groups validation

Getting TPR and FPR: in determining test score sensitivity and specificity

We depend on researchers to tell us what the estimates

of true positive rate and false positive rate are.

They usually do this through a process called

“criterion groups validation.”

People with more confidence than might be called

for refer to this process as “known groups validation.”


Solving classification problems for symptom validity tests with mixed groups validation

The process is seemingly straightforward. in determining test score sensitivity and specificity

Identify two groups. One group has the condition.

All the positives in this group are “true positives.”

One group doesn’t have the condition. All the positives

in this group are “false positives.”

The rate of “true positives” is the sensitivity of the test.

TPR = sensitivity.

The rate of “false positives” is the non-specificity of

the test. FPR = 1 – specificity.


Solving classification problems for symptom validity tests with mixed groups validation

There are many problems with this process, but let’s in determining test score sensitivity and specificity

focus on the main two.

Problem 1

In Study 1, for a given cut score, researchers report the

TPR is .67 and the FPR is .12.

In Study 2, for the same cut score, researchers report

TPR = .58 and FPR = .09.

Which values do you use?


Solving classification problems for symptom validity tests with mixed groups validation

Problem 2: in determining test score sensitivity and specificity

In Study 1, for a given cut score, researchers report the

TPR is .67 and the FPR is .12.

In Study 2, for a different cut score, researchers report

TPR = .58 and FPR = .09.

Which cut score do you use?


Known groups validation
“Known” groups validation in determining test score sensitivity and specificity


Let s validate a test
Let’s validate a test! in determining test score sensitivity and specificity



We take our best shot at a test
We take our best shot at a test. responders and 100 feigners.

TRUTH

TEST


Test results
Test results responders and 100 feigners.

TRUTH

TEST


We say for our test true positive rate 49 100 49 sens 49 false positive rate 1 100 1 specificity 99
We say for our test: responders and 100 feigners.True positive rate = 49/100 = 49% [sens = 49%]False positive rate = 1/100 = 1% [specificity = 99%]

TRUTH

TEST


Solving classification problems for symptom validity tests with mixed groups validation

Because God does not whisper to us anymore, responders and 100 feigners.we take this test, our best test, and we say, “This is the best we can do.” Let’s call it our Gold Standard.We will now make criterion groups with this test,and we will call the groups “Known Groups.”We will then validate tests, based on these Known Groups.


Solving classification problems for symptom validity tests with mixed groups validation
We say for our test: responders and 100 feigners.True positive rate = 49/100 = 49% [sensitivity = 49%]False positive rate = 1/100 = 1% [specificity = 99%]

TRUTH

TEST


Our move from truth to known groups
Our move from TRUTH to KNOWN GROUPS responders and 100 feigners.

“KNOWN” GROUPS

TRUTH



Solving classification problems for symptom validity tests with mixed groups validation

Let’s validate a new test, which standardjust happens to be a perfect test. What test diagnostic efficiencies will we assign our new, perfect, test?

“KNOWN” GROUPS

PERFECT TEST


Solving classification problems for symptom validity tests with mixed groups validation

Let’s validate a new test, which standardjust happens to be a perfect test. What test diagnostic efficiencies will we assign our new, perfect, test?

“KNOWN” GROUPS

Our belief that

we can make

perfect criterion

groups from

imperfect criteria

has led us to

misunderstand

tremendously

what we are

doing.

PERFECT TEST

TPR = 49/50 = 98%, FPR = 51/150 = 34%


Solving classification problems for symptom validity tests with mixed groups validation

Let’s begin to address these problems in a standard

non-traditional way.








Solving classification problems for symptom validity tests with mixed groups validation

REMINDER: Here is what we are working on— standardfiguring out

which positives in our clinic are true positives.

First, we estimate what the true positive rate of the cut

score is. Then, we estimate what the false positive rate of

the cut score is.

Let’s do that part now.

Then, we figure out what percentage of people in our

sample are feigning.

Then we can get the ratio of the mixture of our true

positive and false positives in all the positive scores in our

clinic.





Solving classification problems for symptom validity tests with mixed groups validation

When BR = 0, 10% of scores standard

positive, all false positives


Solving classification problems for symptom validity tests with mixed groups validation

When we standard

say FPR = .16

and TPR = .50,

what we’re

saying is that,

no matter

what samples

we test, we

expect to

see no fewer

than 16%

positive scores

and no more

than 50%

positive

scores.

Movement along this line from left to right

represents increasing rate of Population A and

increasing rate of positive scores.


Solving classification problems for symptom validity tests with mixed groups validation

FPR = .10 standard

TPR = .80



Solving classification problems for symptom validity tests with mixed groups validation

TOMM standard

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093


Solving classification problems for symptom validity tests with mixed groups validation

For any standard

imperfect

test,

PPP ranges

from 0 to 1

as base rate

ranges from

0 to 1

NPP ranges

from 0 to 1

as base rate

ranges from

1 to 0

NPP

PPP


Using mgv to estimate test diagnostic efficiencies of the reliable digit span
Using MGV to estimate standardtest diagnostic efficiencies of the Reliable Digit Span

Laurie Ragatz, PhD

Richard Frederick, PhD


What is reliable digit span
What is Reliable Digit Span? standard

RDS is a symptom validity measure for Digit Span. The value of RDS is derived by adding longest strings of two trials passed for both

forward and backward Digit Span.

Researched cut scores include 5 or lower, 6 or lower, 7 or lower, or 8 or lower.


Reliable digit span example
Reliable Digit Span Example: standard

Directions: Examinee recalls numbers in the same order they were provided by the examiner

Directions: Examinee recalls numbers in the reverse order they were provided by the examiner

Reliable Digit Span: 4 + 3 =7


Solving classification problems for symptom validity tests with mixed groups validation

(1) We found all available articles dealing standard

with RDS and identified the cut scores

investigated. We included simulator studies.

(2) Based on the authors’ decision about

criterion group membership, we

calculated the overall base rate of

malingering in the study.

(3) We observed the overall rate of

positive scores in the study at the

identified cut score.


Solving classification problems for symptom validity tests with mixed groups validation

(4) We did not include any data for persons standard

with mental retardation. The rate of

positive scores among persons with mental

retardation was exceedingly high for all cut

scores.


Solving classification problems for symptom validity tests with mixed groups validation

Criterion group standard

Test outcome

We have 63 malingerers in a sample of 203. BR = 63/203 = 0.31.

We have 57 positive scores. Proportion positive scores (PPS) is 57/203 = .28. For this study, we plot (BR, PPS) = (.31, .28)

x = .31, y = .28. Our n for WLS = 203.


Rds 5 or lower
RDS = 5 or lower standard

Using weighted least squares regression (with N as the weight),

we regressed Proportion Positive Scores (PPS) on Base Rate (BR)

to generate the Proportion Positive Score Line.

We obtained y-intercept of -.015 (all negative values are

truncated to 0), and slope of .265.


Solving classification problems for symptom validity tests with mixed groups validation

RDS = 5 or lower standard

put these data in WLS

to obtain regression

line characteristics

scatterplot



Solving classification problems for symptom validity tests with mixed groups validation

RDS = 6 or lower standard

y-intercept = .015, slope = .419



Solving classification problems for symptom validity tests with mixed groups validation

RDS = 7 or lower standard

y-intercept = .187, slope = .39



Solving classification problems for symptom validity tests with mixed groups validation

RDS = 8 or lower standard

y-intercept = .236, slope = .824



Solving classification problems for symptom validity tests with mixed groups validation

As we move standard

from a cut score

of 5 or lower to 6

or lower, we

obtain

substantial

improvement in

TPR estimate

with little cost

in FPR increase.


Solving classification problems for symptom validity tests with mixed groups validation

Our choice for best cut score for RDS standard

RDS: 6 or lower, FPR = .015, TPR = .434


Solving classification problems for symptom validity tests with mixed groups validation

By using WLS regression, we can obtain standard errors of standard

our estimates of FPR and TPR.

So, new researchers can test hypotheses about parametric

values of FPR and TPR.


Solving classification problems for symptom validity tests with mixed groups validation
Overcoming limiting factors of “known groups” validation in determining test score sensitivity and specificity


Solving classification problems for symptom validity tests with mixed groups validation

  • Summary: in determining test score sensitivity and specificity

  • The TVS and MGV allow powerful research into existing published data sets. Summary data are used.

  • Understanding of parametric values of TPR and FPR is facilitated when researchers publish results on a variety of cut scores that should be considered. A frequency

  • distribution would be ideal, for example,


Solving classification problems for symptom validity tests with mixed groups validation

  • Combining studies in this way allows us to generate in determining test score sensitivity and specificity

  • stable values of TPR and FPR with SE’s so that new research

  • can test those values.

  • Researchers should focus on the basis for estimating BR’s

  • in their research groups. All research estimating FPR and TPR

  • is vulnerable to error when the purity of research groups is

  • overestimated. Working towards a reliable estimate of

  • mixed group base rate will facilitate better validation studies.


Solving classification problems for symptom validity tests with mixed groups validation
Reliably estimate local base rates of feigning for proper allocation of sensitivity and specificity information


Solving classification problems for symptom validity tests with mixed groups validation

  • How can the Test Validation Summary help me allocation of sensitivity and specificity information

  • determine my local BR?

  • Get the best estimate of the test FPR and TPR

  • for a certain test score.

  • 2. Find the proportion of test scores in your

  • sample that are positive scores.


Solving classification problems for symptom validity tests with mixed groups validation

From a sample, observe allocation of sensitivity and specificity information

rate of positive scores.

Use TVS to estimate

condition BR in that

sample, PPP and NPP

for that BR.

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077


Solving classification problems for symptom validity tests with mixed groups validation

TOMM allocation of sensitivity and specificity information

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093


Solving classification problems for symptom validity tests with mixed groups validation

Beth A. Caillouet, Bernice A. Marcopulos, Jesse G. Brand, allocation of sensitivity and specificity information

Julie Ann Kent, & Richard I. Frederick

Question: What are the BRs of malingering in the two samples?


Solving classification problems for symptom validity tests with mixed groups validation

Question: What are the BRs of malingering in the two samples?

Information needed:

Estimates of TOMM FPR and TPR. From TOMM TVS, we

get FPR = .056, TPR = 742.

Sample 1: Secondary gain present. Proportion positive

scores = 55/220 = .25.

Sample 2: Secondary gain absent. Proportion positive

scores = 34/299 = .11.

Use TOMM TVS to estimate BR of each sample.


Solving classification problems for symptom validity tests with mixed groups validation

When PPS = .25, samples?

BR = .28.

When PPS = .11,

BR = .08.


Defensibly choose symptom validity cut scores that are ideally suited for their local base rates
Defensibly choose symptom validity cut scores that are ideally suited for their local base rates


M fast
M-FAST ideally suited for their local base rates


Solving classification problems for symptom validity tests with mixed groups validation

TPR = .93FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


Solving classification problems for symptom validity tests with mixed groups validation

TPR = .93FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


Solving classification problems for symptom validity tests with mixed groups validation

TPR = .93FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


Solving classification problems for symptom validity tests with mixed groups validation

TPR = .93FPR = .17 ideally suited for their local base rates

BR malingering = 35%, N = 86


Solving classification problems for symptom validity tests with mixed groups validation

TPR = .93FPR = 10/56 = .18 ideally suited for their local base rates

BR malingering = 35%, N = 86


Solving classification problems for symptom validity tests with mixed groups validation

TPR = 28/30 = .93FPR = 10/56 = .18 ideally suited for their local base rates

PPP = 28/38 = .737NPP = .958

BR malingering = .35

NPP PPP 1 – FPR TPR


Solving classification problems for symptom validity tests with mixed groups validation

TPR = 28/30 = .93FPR = 10/56 = .18 ideally suited for their local base rates

PPP = 28/131 = .213NPP = .996

BR malingering = .05

NPP PPP 1 – FPR TPR


Solving classification problems for symptom validity tests with mixed groups validation

Test validation ideally suited for their local base rates

summary for

M-FAST cut

score

recommended

by test manual.

PPP does not even reach 50%

correct decisions until BR > .16

M-FAST > 5

FPR = .17

TPR = .93

At recommended cut score FPR very high


Solving classification problems for symptom validity tests with mixed groups validation

At BR = .05, PPP does not exceed ideally suited for their local base rates

.50 until cut score adjusted to

> 9 on M-FAST


Solving classification problems for symptom validity tests with mixed groups validation

Combining information from local base ideally suited for their local base rates

rate and multiple symptom validity tests

You can get estimates of PPP and NPP

for the sample you work with—IF you

can reliably estimate the BR.


Solving classification problems for symptom validity tests with mixed groups validation

737 defendants were administered: ideally suited for their local base rates

Rey 15 Item Memory Test (RMT)—memorize and

reproduce 15 items—very easy test.

Score is items reproduced (0 to 15)

Word Recognition Test (WRT)—memorize 15 words,

identify those 15 and correctly reject 15 from a list of 30.

Score is number of hits and correct rejections (0 to 30)


Solving classification problems for symptom validity tests with mixed groups validation

RMT validating ideally suited for their local base rates

using MGV

with clinical

probability

judgments.

FPR = .025

TPR = .574

Frederick & Bowden,

2009


Solving classification problems for symptom validity tests with mixed groups validation

RMT < 9 ideally suited for their local base rates

FPR = .025

TPR = .574

We found 726 defendants who completed BOTH RMT and WRT.

81/726 failed the RMT= .111 proportion positive score.

By observation of TVS, then BR = .16, PPP = .814, NPP = .923


Solving classification problems for symptom validity tests with mixed groups validation

From a sample, observe ideally suited for their local base rates

rate of positive scores.

Use TVS to estimate

condition BR in that

sample, PPP and NPP

for that BR.

527 criminal

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077


Solving classification problems for symptom validity tests with mixed groups validation

We found 726 defendants who completed BOTH RMT and WRT. ideally suited for their local base rates

81/726 failed the RMT= .111 proportion positive score.

By observation of TVS, then BR = .16, PPP = .814, NPP = .923

If PPP = .814, then in this sample, the probability of feigning if RMT

is positive, is .814.

If NPP = .923, then in this sample, the probability of feigning if RMT

is negative is .077,

or 1 - .923.

  • To conduct MGV, we sampled from two groups:

  • The 645 individuals who passed the RMT—had a negative score.

  • The 81 individuals who failed the RMT—had a positive score.


Solving classification problems for symptom validity tests with mixed groups validation

Example of sampling ideally suited for their local base rates

645 individuals with

negative scores, p(mal) = .077

81 individuals with

positive scores, p(mal) = .814

Sample n = 360

Sample n = 40

400 cases, 10% failures, 90% passes

Overall p(mal) = 40*.814 + 360*.077 = .151

Sample 25 times, plot x = .151, y = observed rate of

positive WRT scores, n for WLS = 400


Solving classification problems for symptom validity tests with mixed groups validation

For each sample, BR was pre-estimated. Then we observed ideally suited for their local base rates

rate of positive WRT scores at each potential cut score.


Solving classification problems for symptom validity tests with mixed groups validation

Word Recognition Test (WRT) ideally suited for their local base rates

Range 4 to 30, Mean = 23.2

Within group of RMT < 9, mean = 18.7

Within group of RMT > 8, mean = 23.8


Solving classification problems for symptom validity tests with mixed groups validation

Word Recognition Test (WRT) ideally suited for their local base rates

For every potential cut score of WRT (4 -30), we plotted all x, y pairs

obtained from sampling

We performed WLS to obtain the FPR and TPR estimates

at every potential cut score.

We plotted the FPR and TPR estimates at every potential cut

score to generate the ROC curve.

AUC = .905, SE = .012, 95% CI for AUC = .881-.930.

Best cut scores:

LTE 18 (TPR = .563, FPR = .034)

LTE 19 (TPR = .620, FPR = .066)


Solving classification problems for symptom validity tests with mixed groups validation

We plotted the FPR and TPR estimates at every potential cut ideally suited for their local base rates

score to generate the ROC curve.

AUC = .905, SE = .012, 95% CI for AUC = .881-.930.

Best cut scores:

LTE 18

(TPR = .563, FPR = .034)

LTE 19

(TPR = .620, FPR = .066)

TPR

FPR

WORD RECOGNITON TEST (WRT)


Solving classification problems for symptom validity tests with mixed groups validation

  • Summary: ideally suited for their local base rates

  • We can use tests to form mixed groups for validation.

  • The best estimates of FPR and TPR for a test cut score

  • allow us to estimate PPP and NPP at our sample BR.

  • Instead of “known groups” design (which is misleading),

  • we do not presume to know (or care) about the status of

  • any individual. We assign individuals “probabilities of

  • having the condition” based on their test score.

  • Mixed groups have an overall “probability of having the

  • condition,” which is the average of the individual probabilities.

  • We do not need to be certain about group memberships.

  • We gain much flexibility by working with probabilities of having

  • the condition vs. certainties of having the condition.


Another example
Another example ideally suited for their local base rates


Solving classification problems for symptom validity tests with mixed groups validation

Dawes 1967 showed that valid probability judgments are ideally suited for their local base rates

excellent base rate indicators. His work was substantiated in Frederick 2000 and Frederick and Bowden 2009.

To conduct MGV, we formed groups of defendants for whom individuals ratings of likelihood of malingering psychosis were generated by forensic psychologists, before any testing took place.

The BR of malingered psychosis for each group was then the mean of the probability rating. If each member of the group had been rated as 10% likely to feign psychosis, then the BR of the group was estimated to be 10%.


Solving classification problems for symptom validity tests with mixed groups validation

We then observed the hit rate (proportion positive scores) for

the groups for a variety of F-family indicators of feigning on the

MMPI-2 and MMPI-2-RF.

We formed 15 groups of 30 individuals. For each group, we

had a static base rate, which was the mean of the probability

judgments assigned before testing.

Within each group, we iteratively observed the hit rate of positive

F-family indicators at each potential cut score. Using the BR

estimate and the proportion positive scores at each potential cut

score, we performed WLS to generate estimates of FPR and TPR.

From these estimates, we generated ROC curves.


Solving classification problems for symptom validity tests with mixed groups validation

15 groups, 30 defendants in each group, 450 defendants for

Each defendant rated from 0 to 100 before testing, with respect

to likelihood he would feign psychosis.

Groups were formed after first sorting individuals by ratings, from

lowest to highest.

Mean ratings of groups (each group, n = 30):

001.24.25.0

5.05.05.08.110

14.522.230.345.772.3

Rates of positive F-family scores at each potential cut observed.


Solving classification problems for symptom validity tests with mixed groups validation

Estimates by Nicholson, Mouton, Bagby, Buis, Peterson, for

and Buigas (1998):

AUC’s and SE: F (.929, .021) Fp (.885, .027)


Solving classification problems for symptom validity tests with mixed groups validation

  • Summary: for

  • Using the estimates of likelihood of feigning based only

  • on clinician judgment prior to testing did not result in

  • random results. We can assume that mean probability

  • judgments were effective base rate estimates.

  • Our estimates of F and Fp are consistent with estimates

  • in large, well-validated analysis.

  • In this study, MMPI-2-RF indicators have higher mean

  • AUC and lower SE than their MMPI-2 counterparts.



Solving classification problems for symptom validity tests with mixed groups validation

f for

FrequencyPercentValid PercentCumulative Percent

Valid442.73.13.1

61.7.83.8

91.7.84.6

1021.31.56.1

1132.02.38.4

1221.31.59.9

1332.02.312.2

1442.73.115.3

1521.31.516.8

1632.02.319.1

1732.02.321.4

1821.31.522.9

1942.73.126.0

2053.43.829.8

2121.31.531.3

2274.75.336.6

2332.02.338.9

241.7.839.7

2553.43.843.5

2664.04.648.1

2764.04.652.7

2874.75.358.0

2921.31.559.5

3021.31.561.1

3164.04.665.6

3253.43.869.5

3332.02.371.8

3442.73.174.8

3542.73.177.9

3632.02.380.2

3774.75.385.5

3842.73.188.5

3921.31.590.1

4032.02.392.4

4132.02.394.7

431.7.895.4

451.7.896.2

461.7.896.9

471.7.897.7

4821.31.599.2

521.7.8100.0

Total13187.9100.0

MissingSystem1812.1

Total149100.0

131 defendants

who took MMPI

and SIRS


Solving classification problems for symptom validity tests with mixed groups validation

52.7% of cases are 27 or lower for

47.3% of cases are 28 or higher

What is the base rate of feigned psychopathology?


Solving classification problems for symptom validity tests with mixed groups validation

BR TPR FPR NPP PPP for

What we say:

Within our sample of 131 defendants, the BR of feigned

psychopathology is .73 (NOT .475)

At BR = .73, the PPP of F GTE 28 is .976.

At BR = .73, the NPP of GTE 28 is .492, so

p(feigning if LTE 27) is still .508) (Remember, they’re

being given the SIRS for a reason)


Solving classification problems for symptom validity tests with mixed groups validation

F < 28 for

NPP about .66


Solving classification problems for symptom validity tests with mixed groups validation

F > 27 for



Solving classification problems for symptom validity tests with mixed groups validation

Greve, Bianchini, Love, Brennan, & for Heinly (2006) articulated six separate

groups with increasing base rate of malingering based on formal criteria

for malingering (the Slick criteria) to validate the MMPI-2 Fake Bad Scale

  • No incentive (no evidence of external incentive and no test

  • performance suggestive of malingering; n = 18, mean FBS = 15.4)

  • Incentive (external incentive, but no test performance suggestive

  • of malingering; n = 79, mean FBS = 19.5)

  • Suspect (external incentive and at least one indicator suggestive

  • of malingering; n = 66, mean FBS = 22.7)

  • Statistically Likely (external incentive; at least two indicators

  • suggestive of malingering; n = 51, mean FBS = 22.8)

  • Probable (external incentive; strong indicators of malingering;

  • n = 31, mean FBS = 26.9)

  • Definite (external incentive; very strong indicators of

  • malingering; n = 14, mean FBS = 29.8)


Solving classification problems for symptom validity tests with mixed groups validation

  • Even though it is clear that for

  • BR Definite > BR Probable > BR Statistically Likely >

  • BR Suspect > BR Incentive Only > BR No Incentive

  • They were required, to conduct “Known” groups

  • validation, to ignore this obvious circumstance and to define

  • BR No Incentive = BR Incentive Only = 0

  • BR Statistically Likely = BR Probable = BR Definite = 1.0

  • And drop all participants defined as Suspect

  • to yield the following ROC


Solving classification problems for symptom validity tests with mixed groups validation

FBS ROC for

generated

by “Known”

groups

validation by

Greve & Bianchini


Solving classification problems for symptom validity tests with mixed groups validation

If we had estimates of the BR for each of the subgroups for

formed by Greve and Bianchini, we could use MGV to

estimate FPR and TPR for each potential cut score.


Solving classification problems for symptom validity tests with mixed groups validation

We have our stable estimate of for

TOMM FPR and TPR


Solving classification problems for symptom validity tests with mixed groups validation

TOMM for

No simulation

studies

FPR = .056,

SE = .025

TPR = .742,

SE = .093


Solving classification problems for symptom validity tests with mixed groups validation

We can get estimates of BRs for those groups for

from other work by Greve & Bianchini.

They formed similar groups using the Slick

criteria to investigate the TOMM.

We can use the proportion of positive TOMMs

in each of these subgroups to estimate the BRs

in each of them.



Solving classification problems for symptom validity tests with mixed groups validation

We take these BR estimates and reapply them for

to the Greve & Bianchini FBS data.


Solving classification problems for symptom validity tests with mixed groups validation

Example of MGV for FBS based on BR estimates for for

Greve & Bianchini groups established by Slick criteria

At FBS > 27

For FBS > 27, using WLS Regression, FPR = .091, TPR = .773

(For WLS, n is the weighted variable)


Evaluate constructs that underlie symptom validity tests
Evaluate constructs that underlie for symptom validity tests


Solving classification problems for symptom validity tests with mixed groups validation

10 clinical for

studies

using Rey

15-Item Test

No simulators

All clinical data


Solving classification problems for symptom validity tests with mixed groups validation

RMT validating for

using MGV

with clinical

probability

judgments.

FPR = .025

TPR = .574

Frederick & Bowden,

2009


Solving classification problems for symptom validity tests with mixed groups validation

CI, TPR = .574, for

SE = .044

We will

generate TVS

based on these

values and

find PPP and

1 – NPP to

estimate

probability of

bad intent

represented

by RMT score.



Vip verbal subtest items
VIP Verbal Subtest Items for

Easy:BabyDrinkInfant

Moderate:PeopleAllyFolk

Difficult:NimietyConceitSurfeit



Solving classification problems for symptom validity tests with mixed groups validation

Not guessing, for

knowledgeable

responding

Guessing

Guessing

is imminent

Easy items

Difficult items



Solving classification problems for symptom validity tests with mixed groups validation

527 criminal for

defendants

who took

RMT and VIP

concurrently

Rate of positive

scores in this

sample was .113

PPP = .814

1 – NPP = .077


Solving classification problems for symptom validity tests with mixed groups validation

Here we for

are matching

VIP categories

to the construct

most likely

captured by

the VIP.

Points in

scatterplot

represent

groups of 25

individuals.

Sorted defendants

by clinical ratings

of malingering,

then took 20

groups of 25 and

one group of 27,

for 527 defendants.

BR of .42 estimated

for this group is mean

of PPP for positive RMT

scores in this group and

(1 – NPP) for negative

RMT scores in the group


Solving classification problems for symptom validity tests with mixed groups validation

Same 21 for

subgroups,

N = 527

defendants


Solving classification problems for symptom validity tests with mixed groups validation

527 criminal for

defendants

VRIN was converted

to “probability

of invalid

responding” by

dividing VRIN

raw score by 12.

VRIN raw scores

>12 were

assigned p = 1.

We are interested

in FPR and TPR for

“Invalid”


ad
  • Login