Circular analysis in systems neuroscience
Download
1 / 62

- PowerPoint PPT Presentation


  • 157 Views
  • Uploaded on

Circular analysis in systems neuroscience – with particular attention to cross-subject correlation mapping. Nikolaus Kriegeskorte Laboratory of Brain and Cognition, National Institute of Mental Health. Collaborators. Chris I Baker W Kyle Simmons Patrick SF Bellgowan Peter Bandettini.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - nairi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Circular analysis in systems neuroscience– with particular attention to cross-subject correlation mapping

Nikolaus Kriegeskorte

Laboratory of Brain and Cognition, National Institute of Mental Health


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Collaborators

Chris I Baker

W Kyle Simmons

Patrick SF Bellgowan

Peter Bandettini


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Overview

Part 1General introduction to circular analysis in systems neuroscience(synopsis of Kriegeskorte et al. 2009)

Part 2Specific issue: selection bias incross-subject correlation mapping(following up on Vul et al. 2009)


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

data

results


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

analysis

data

results


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

analysis

data

results


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

assumptions

analysis

data

results


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

assumptions

data

results

analysis


Circular inference
Circular inference

assumptions

analysis

data

results


Circular inference1
Circular inference

assumptions

analysis

data

results


How do assumptions tinge results

Weighting

(continuous selection)

Elimination

(binary selection)

Sorting

(multiclass selection)

How do assumptions tinge results?

– Through variants of selection!


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Elimination

(binary selection)

assumptions:

selection criteria

analysis

data

results


Example 1 pattern information analysis

Example 1Pattern-information analysis


Experimental design
Experimental design

TASK

(property judgment)

Simmons et al. 2006

“Animate?”

“Pleasant?”

STIMULUS

(object category)


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Pattern-information analysis

define ROI by selecting ventral-temporal voxels for which any pairwise condition contrast is significant at p<.001 (uncorr.)

perform nearest-neighbor classificationbased on activity-pattern correlation

use oddruns for trainingand evenruns for testing


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Results

stimulus

(object category)

task

(judged property)

decoding accuracy

chance level

1

0.5

0


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

stimulus

task

decoding accuracy

chance level

!

?

fMRI data

data from Gaussian

random generator

using all data

to select ROI voxels

1

1

1

1

...but we used cleanly independent

training and test data!

using only

training data

to select ROI voxels

0.5

0.5

0.5

0.5

0

0

0

0


Conclusion for pattern information analysis
Conclusion for pattern-information analysis

The test data must not be used in either...

  • training a classifier or

  • defining the ROI

continuous weighting

binary weighting


Data selection is key to many conventional analyses

Data selection is key to many conventional analyses.

Can it entail similar biases in other contexts?


Example 2 regional activation analysis

Example 2Regional activation analysis


Roi definition is affected by noise
ROI definition is affected by noise

independent

ROI

overfitted

ROI

true region

overestimated effect

ROI-average

activation


Data sorting
Data sorting

assumptions:

sorting criteria

analysis

data

results


Set average tuning curves
Set-average tuning curves

...for data sorted by tuning

response

stimulus parameter (e.g. orientation)

noise data


Set average activation profiles

ROI-average

fMRI response

A

B

C

D

condition

Set-average activation profiles

...for data sorted by activation

noise data


To avoid selection bias we can
To avoid selection bias, we can...

...perform a nonselective analysis

OR

...make sure that selection and results statistics are independent under the null hypothesis,

because they are either:

  • inherently independent

  • or computed on independent data

e.g. whole-brain mapping

(no ROI analysis)

e.g. independent contrasts


Does selection by an orthogonal contrast vector ensure unbiased analysis
Does selection by an orthogonal contrast vector ensure unbiased analysis?

cselection=[1 1]T

ctest=[1 -1]T

orthogonal contrast vectors 

ROI-definition contrast: A+B

ROI-average analysis contrast: A-B


Does selection by an orthogonal contrast vector ensure unbiased analysis1
Does selection by an orthogonal contrast vector ensure unbiased analysis?

contrast

vector

– No, there can still be bias.

still not sufficient

not sufficient

The design and noise dependencies matter.

design

noise dependencies


Circular analysis
Circular analysis unbiased analysis?

Pros

Cons

  • highly sensitive

  • widely accepted (examples in all high-impact journals)

  • doesn't require independent data sets

  • grants scientists independencefrom the data

  • allows smooth blending of blind faith and empiricism


Circular analysis1
Circular analysis unbiased analysis?

Pros

Cons

  • highly sensitive

  • widely accepted (examples in all high-impact journals)

  • doesn't require independent data sets

  • grants scientists independencefrom the data

  • allows smooth blending of blind faith and empiricism


Circular analysis2
Circular analysis unbiased analysis?

Pros

Pros

Cons

[can’t think of any right now]

  • the error that beautifies results

  • confirms even incorrect hypotheses

  • improves chances ofhigh-impact publication

  • highly sensitive

  • widely accepted (examples in all high-impact journals)

  • doesn't require independent data sets

  • grants scientists independencefrom the data

  • allows smooth blending of blind faith and empiricism


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Part 2 unbiased analysis?Specific issue: selection bias incross-subject correlation mapping(following up on Vul et al. 2009)


Motivation
Motivation unbiased analysis?

Vul et al. (2009) posed a puzzle:

Why are the cross-subject correlations found in brain mapping so high?

Selection bias is one piece of the puzzle.

But there are more pieces and we have yet to put them all together.


Overview
Overview unbiased analysis?

  • List and discuss six pieces of the puzzle.

    (They don't all point in the same direction!)

  • Suggest some guidelines for good practice.


Six pieces synopsis
Six pieces synopsis unbiased analysis?

  • Cross-subject correlation estimates are very noisy.

  • Bin or within-subject averaging legitimately increases correlations.

  • Selecting among noisy estimates yields large biases.

  • False-positive regions are highly likely for a whole-brain mapping thresholded at p<.001, uncorrected.

  • Reported correlations are high, but not highly significant.

  • Studies have low power for finding realistic correlations in the brain if multiple testing is appropriately accounted for.


Vul et al 2009
Vul et al. 2009 unbiased analysis?

,,

noise-free

correlation

population

,,

The geometric mean of the reliability is an upper bound

on the population correlation.

The reliabilities provide no bound

on the sample correlation.


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Piece 1 unbiased analysis?

Sample correlationsacross small numbers of subjectsare very noisy estimatesof population correlations.


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

0.65 unbiased analysis?


Cross subject correlation estimates are very noisy
Cross-subject correlation estimates unbiased analysis?are very noisy

95%-confidence

interval

correlation

10 subjects


Cross subject correlation estimates are very noisy1
Cross-subject correlation estimates unbiased analysis?are very noisy


The more we average reducing noise but not signal the higher correlations become

Piece 2 unbiased analysis?

The more we average(reducing noise but not signal),the higher correlations become.




Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Subjects are like bins unbiased analysis?...

For each subject, all data is averaged to give one number.

Take-home message

Cross-subject correlation estimates are expected to be...

  • high (averaging all data for each subject)

  • noisy (low number of subjects)

So what's Ed fussing about?We don't need selection bias to explain the high correlations, right?


Selecting the maximum among noisy estimates yields large selection biases

Piece 3 unbiased analysis?

Selecting the maximumamong noisy estimatesyields large selection biases.


Expected maximum correlation selected among null regions
Expected maximum correlation unbiased analysis?selected among null regions

expected maximum correlation

bias

16 subjects


False positive regions are likely to be found in whole brain mapping using p 001 uncorrected

Piece 4 unbiased analysis?

False-positive regions are likely to be found in whole-brain mappingusing p<.001, uncorrected.


Mapping with p 001 uncorrected
Mapping with p<.001, uncorrected unbiased analysis?

Global null hypothesis is true

(population correlation = 0 in all brain locations)


Reported correlations are high but not highly significant

Piece 5 unbiased analysis?

Reported correlations are high,but not highly significant.


Reported correlations are high but not highly significant1

correlation thresholds as a function unbiased analysis?

of the number of subjects

one-sided

two-sided

Reported correlations are high,but not highly significant

p<0.00001

p<0.001p<0.01

p<0.05


Reported correlations are high but not highly significant2

correlation thresholds as a function unbiased analysis?

of the number of subjects

one-sided

two-sided

Reported correlations are high,but not highly significant

p<0.00001

p<0.001p<0.01

p<0.05


Reported correlations are high but not highly significant3

correlation thresholds as a function unbiased analysis?

of the number of subjects

one-sided

two-sided

Reported correlations are high,but not highly significant

What correlations would we expect

under the global null hypothesis?

(assuming each study reports

the maximum of 500

independent brain locations)

p<0.00001

p<0.001p<0.01

p<0.05


Reported correlations are high but not highly significant4

one-sided unbiased analysis?

two-sided

Reported correlations are high,but not highly significant

What correlations would we expect

under the global null hypothesis?

p<0.00001

p<0.001p<0.01

p<0.05

(assuming each study reports the max.of 500 independent brain locations)


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Piece 6 unbiased analysis?

Most of the studies have low powerfor finding realistic correlationswith whole-brain mappingif multiple testing is appropriately accounted for.

see also: Yarkoni 2009


Numbers of subjects in studies reviewed by vul et al 2009
Numbers of subjects unbiased analysis?in studies reviewed by Vul et al. (2009)

number of correlations estimates

4

8

16

36

60

100

number of subjects


In order to find a single region with a cross subject correlation of 0 7 in the brain

power unbiased analysis?

In order to find a single region with across-subject correlation of 0.7 in the brain...

...we would need

about 36 subjects

16 subjects


In order to find a single region with a cross subject correlation of 0 7 in the brain1

power unbiased analysis?

In order to find a single region with across-subject correlation of 0.7 in the brain...

...we would need

about 36 subjects

16 subjects


Circular analysis in systems neuroscience with particular attention to cross subject correlation mapping

Take-home message unbiased analysis?

Whole-brain cross-subject correlation mapping

with 16 subjects

does

not

work.

Need at least twice as many subjects.


Conclusions
Conclusions unbiased analysis?

Unless much larger numbers of subjects are used, whole-brain cross-subject correlation mapping suffers from either:

  • very low power to detect true regions(if we carefully to correct for multiple comparisons)

  • very high rates of false-positive regions(otherwise)

    If analysis is circular, selection bias is expected to be high here (because selection occurs among noisy estimates).

...in other words,

it doesn't work.


Suggestions
Suggestions unbiased analysis?

  • Design study to have enough power to detect realistic correlations. (Need either anatomical restrictions or large numbers of subjects.)

  • Consider studying trial-to-trial rather than subject-to-subject effects.

  • Correct for multiple testing to avoid false positives.

  • Avoid circularity: Use leave-one-subject out procedure to estimate regional cross-subject correlations.

  • Report correlation estimates with error bars.