slide1
Download
Skip this Video
Download Presentation
Circular analysis in systems neuroscience – with particular attention to cross-subject correlation mapping

Loading in 2 Seconds...

play fullscreen
1 / 62

Circular analysis in systems neuroscience with particular attention to cross-subject correlation mapping - PowerPoint PPT Presentation


  • 157 Views
  • Uploaded on

Circular analysis in systems neuroscience – with particular attention to cross-subject correlation mapping. Nikolaus Kriegeskorte Laboratory of Brain and Cognition, National Institute of Mental Health. Collaborators. Chris I Baker W Kyle Simmons Patrick SF Bellgowan Peter Bandettini.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Circular analysis in systems neuroscience with particular attention to cross-subject correlation mapping' - nairi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Circular analysis in systems neuroscience– with particular attention to cross-subject correlation mapping

Nikolaus Kriegeskorte

Laboratory of Brain and Cognition, National Institute of Mental Health

slide2

Collaborators

Chris I Baker

W Kyle Simmons

Patrick SF Bellgowan

Peter Bandettini

slide3

Overview

Part 1General introduction to circular analysis in systems neuroscience(synopsis of Kriegeskorte et al. 2009)

Part 2Specific issue: selection bias incross-subject correlation mapping(following up on Vul et al. 2009)

slide5

data

results

slide6

analysis

data

results

slide7

analysis

data

results

slide8

assumptions

analysis

data

results

slide9

assumptions

data

results

analysis

circular inference
Circular inference

assumptions

analysis

data

results

circular inference1
Circular inference

assumptions

analysis

data

results

how do assumptions tinge results

Weighting

(continuous selection)

Elimination

(binary selection)

Sorting

(multiclass selection)

How do assumptions tinge results?

– Through variants of selection!

slide13

Elimination

(binary selection)

assumptions:

selection criteria

analysis

data

results

experimental design
Experimental design

TASK

(property judgment)

Simmons et al. 2006

“Animate?”

“Pleasant?”

STIMULUS

(object category)

slide16

Pattern-information analysis

define ROI by selecting ventral-temporal voxels for which any pairwise condition contrast is significant at p<.001 (uncorr.)

perform nearest-neighbor classificationbased on activity-pattern correlation

use oddruns for trainingand evenruns for testing

slide17

Results

stimulus

(object category)

task

(judged property)

decoding accuracy

chance level

1

0.5

0

slide18

stimulus

task

decoding accuracy

chance level

!

?

fMRI data

data from Gaussian

random generator

using all data

to select ROI voxels

1

1

1

1

...but we used cleanly independent

training and test data!

using only

training data

to select ROI voxels

0.5

0.5

0.5

0.5

0

0

0

0

conclusion for pattern information analysis
Conclusion for pattern-information analysis

The test data must not be used in either...

  • training a classifier or
  • defining the ROI

continuous weighting

binary weighting

data selection is key to many conventional analyses

Data selection is key to many conventional analyses.

Can it entail similar biases in other contexts?

roi definition is affected by noise
ROI definition is affected by noise

independent

ROI

overfitted

ROI

true region

overestimated effect

ROI-average

activation

data sorting
Data sorting

assumptions:

sorting criteria

analysis

data

results

set average tuning curves
Set-average tuning curves

...for data sorted by tuning

response

stimulus parameter (e.g. orientation)

noise data

set average activation profiles

ROI-average

fMRI response

A

B

C

D

condition

Set-average activation profiles

...for data sorted by activation

noise data

to avoid selection bias we can
To avoid selection bias, we can...

...perform a nonselective analysis

OR

...make sure that selection and results statistics are independent under the null hypothesis,

because they are either:

  • inherently independent
  • or computed on independent data

e.g. whole-brain mapping

(no ROI analysis)

e.g. independent contrasts

does selection by an orthogonal contrast vector ensure unbiased analysis
Does selection by an orthogonal contrast vector ensure unbiased analysis?

cselection=[1 1]T

ctest=[1 -1]T

orthogonal contrast vectors 

ROI-definition contrast: A+B

ROI-average analysis contrast: A-B

does selection by an orthogonal contrast vector ensure unbiased analysis1
Does selection by an orthogonal contrast vector ensure unbiased analysis?

contrast

vector

– No, there can still be bias.

still not sufficient

not sufficient

The design and noise dependencies matter.

design

noise dependencies

circular analysis
Circular analysis

Pros

Cons

  • highly sensitive
  • widely accepted (examples in all high-impact journals)
  • doesn\'t require independent data sets
  • grants scientists independencefrom the data
  • allows smooth blending of blind faith and empiricism
circular analysis1
Circular analysis

Pros

Cons

  • highly sensitive
  • widely accepted (examples in all high-impact journals)
  • doesn\'t require independent data sets
  • grants scientists independencefrom the data
  • allows smooth blending of blind faith and empiricism
circular analysis2
Circular analysis

Pros

Pros

Cons

[can’t think of any right now]

  • the error that beautifies results
  • confirms even incorrect hypotheses
  • improves chances ofhigh-impact publication
  • highly sensitive
  • widely accepted (examples in all high-impact journals)
  • doesn\'t require independent data sets
  • grants scientists independencefrom the data
  • allows smooth blending of blind faith and empiricism
slide32

Part 2Specific issue: selection bias incross-subject correlation mapping(following up on Vul et al. 2009)

motivation
Motivation

Vul et al. (2009) posed a puzzle:

Why are the cross-subject correlations found in brain mapping so high?

Selection bias is one piece of the puzzle.

But there are more pieces and we have yet to put them all together.

overview
Overview
  • List and discuss six pieces of the puzzle.

(They don\'t all point in the same direction!)

  • Suggest some guidelines for good practice.
six pieces synopsis
Six pieces synopsis
  • Cross-subject correlation estimates are very noisy.
  • Bin or within-subject averaging legitimately increases correlations.
  • Selecting among noisy estimates yields large biases.
  • False-positive regions are highly likely for a whole-brain mapping thresholded at p<.001, uncorrected.
  • Reported correlations are high, but not highly significant.
  • Studies have low power for finding realistic correlations in the brain if multiple testing is appropriately accounted for.
vul et al 2009
Vul et al. 2009

,,

noise-free

correlation

population

,,

The geometric mean of the reliability is an upper bound

on the population correlation.

The reliabilities provide no bound

on the sample correlation.

cross subject correlation estimates are very noisy
Cross-subject correlation estimatesare very noisy

95%-confidence

interval

correlation

10 subjects

slide46
Subjects are like bins...

For each subject, all data is averaged to give one number.

Take-home message

Cross-subject correlation estimates are expected to be...

  • high (averaging all data for each subject)
  • noisy (low number of subjects)

So what\'s Ed fussing about?We don\'t need selection bias to explain the high correlations, right?

expected maximum correlation selected among null regions
Expected maximum correlationselected among null regions

expected maximum correlation

bias

16 subjects

mapping with p 001 uncorrected
Mapping with p<.001, uncorrected

Global null hypothesis is true

(population correlation = 0 in all brain locations)

reported correlations are high but not highly significant1

correlation thresholds as a function

of the number of subjects

one-sided

two-sided

Reported correlations are high,but not highly significant

p<0.00001

p<0.001p<0.01

p<0.05

reported correlations are high but not highly significant2

correlation thresholds as a function

of the number of subjects

one-sided

two-sided

Reported correlations are high,but not highly significant

p<0.00001

p<0.001p<0.01

p<0.05

reported correlations are high but not highly significant3

correlation thresholds as a function

of the number of subjects

one-sided

two-sided

Reported correlations are high,but not highly significant

What correlations would we expect

under the global null hypothesis?

(assuming each study reports

the maximum of 500

independent brain locations)

p<0.00001

p<0.001p<0.01

p<0.05

reported correlations are high but not highly significant4

one-sided

two-sided

Reported correlations are high,but not highly significant

What correlations would we expect

under the global null hypothesis?

p<0.00001

p<0.001p<0.01

p<0.05

(assuming each study reports the max.of 500 independent brain locations)

slide56

Piece 6

Most of the studies have low powerfor finding realistic correlationswith whole-brain mappingif multiple testing is appropriately accounted for.

see also: Yarkoni 2009

numbers of subjects in studies reviewed by vul et al 2009
Numbers of subjectsin studies reviewed by Vul et al. (2009)

number of correlations estimates

4

8

16

36

60

100

number of subjects

in order to find a single region with a cross subject correlation of 0 7 in the brain

power

In order to find a single region with across-subject correlation of 0.7 in the brain...

...we would need

about 36 subjects

16 subjects

in order to find a single region with a cross subject correlation of 0 7 in the brain1

power

In order to find a single region with across-subject correlation of 0.7 in the brain...

...we would need

about 36 subjects

16 subjects

slide60
Take-home message

Whole-brain cross-subject correlation mapping

with 16 subjects

does

not

work.

Need at least twice as many subjects.

conclusions
Conclusions

Unless much larger numbers of subjects are used, whole-brain cross-subject correlation mapping suffers from either:

  • very low power to detect true regions(if we carefully to correct for multiple comparisons)
  • very high rates of false-positive regions(otherwise)

If analysis is circular, selection bias is expected to be high here (because selection occurs among noisy estimates).

...in other words,

it doesn\'t work.

suggestions
Suggestions
  • Design study to have enough power to detect realistic correlations. (Need either anatomical restrictions or large numbers of subjects.)
  • Consider studying trial-to-trial rather than subject-to-subject effects.
  • Correct for multiple testing to avoid false positives.
  • Avoid circularity: Use leave-one-subject out procedure to estimate regional cross-subject correlations.
  • Report correlation estimates with error bars.
ad