slide1
Download
Skip this Video
Download Presentation
Computational models of cognitive control (II)

Loading in 2 Seconds...

play fullscreen
1 / 184

Computational models of cognitive control (II) - PowerPoint PPT Presentation


  • 70 Views
  • Uploaded on

Computational models of cognitive control (II). Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology, Princeton University. Banishing the homunculus. Banishing the homunculus Decision-making in control:. Banishing the homunculus Decision-making in control:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Computational models of cognitive control (II)' - orde


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University

slide3
Banishing the homunculus

Decision-making in control:

slide4
Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

slide5
Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”

slide6
Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”

And, “How are they updated over time?”

1 routine sequential action

1. Routine sequential action

Botvinick & Plaut, Psychological Review, 2004

Botvinick, Proceedings of the Royal Society, B, 2007.

Botvinick, TICS, 2008

slide9
‘Routine sequential action’
  • Action on familiar objects
  • Well-defined sequential structure
  • Concrete goals
  • Highly routine
  • Everyday tasks
slide10
?!

Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University

slide11
ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARPACK

ADD SUGAR FROM SUGARBOWL

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical structure

MAKE INSTANT COFFEE

slide12
MAKE INSTANT COFFEE

ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARBOWL / PACKET

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical models of action

(e.g.,Cooper & Shallice, 2000; Estes, 1972; Houghton, 1990; MacKay, 1987, Rumelhart & Norman, 1982)

  • Schemas as primitive elements
  • Hierarchical structure of task built directly into architecture
slide13
at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

An alternative approach

slide14
at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

  • p, s, a = patterns of activation over simple processing units
  • Weighted, excitatory/inhibitory connections
  • Weights adjusted through gradient-descent learning in target task domains
slide15
Recurrent neural networks
  • Feedback as well as feedforward connections
  • Allow preservation of information over time
  • Demonstrated capacity to learn sequential behaviors

(e.g., Cleermans, 1993; Elman, 1990)

slide16
environment

The model

action

internal

representation

perceptual input

slide17
Fixate(Blue)

Fixate(Green)

Fixate(Top)

PickUp

Fixate(Table)

PutDown

Fixate(Green)

PickUp

Ballard, Hayhoe, Pook & Rao, (1996). BBS.

slide18
environment

Model architecture

manipulative

perceptual

action

perceptual input

viewed object

held object

slide19
Routine sequential action: Task domain
  • Hierarchically structured
  • Actions/subtasks may appear in multiple contexts
  • Environmental cues alone sometimes insufficient to guide action selection
  • Subtasks that may be executed in variable order
  • Subtask disjunctions
slide20
grounds

cream

cream

Start

drink

End

`

steep tea

drink

End

slide21
Manipulative

actions

sugar-packet

Perceptual

actions

Representations

slide22
Input

Target/

output

slide23
Input

Target/

output

slide24
Input

Target/

output

slide25
Input

Target/

output

slide26
Input

Target/

output

slide27
Input

Target/

output

slide28
Input

Target/

output

slide30
grounds

cream

cream

Start

drink

End

15%

18%

grounds

cream

cream

Start

drink

End

12%

10%

grounds

grounds

cream

cream

cream

cream

Start

Start

drink

drink

End

End

20%

25%

Start

Start

drink

steep tea

drink

steep tea

End

End

slide31
Slips of action

(after Reason)

  • Occur at decision (or fork) points
  • Sequence errors involve subtask omissions, repetitions, and lapses
  • Lapses show effect of relative task frequency
slide32
manipulative perceptual

action

perceptual input

viewed object

held object

environment

slide33
Sample of behavior:
  • pick-up coffee-pack
  • pull-open coffee-pack
  • pour coffee-pack into cup
  • put-down coffee-pack
  • pick-up spoon
  • stir cup
  • put-down spoon
  • pick-up sugar-pack
  • tear-open sugar-pack
  • pour sugar-pack into cup
  • put-down sugar-pack
  • pick-up spoon
  • stir cup
  • put-down spoon
  • pick-up cup*
  • sip cup
  • sip cup
  • say-done

grounds

sugar (pack)

cream omitted

drink

slide34
subtask 1 subtask 2 subtask 3 subtask 4

100

Percentage of trials error-free

0

Step in coffee sequence

slide35
Omissions / anticipations

Repetitions / perseverations

Intrusions / lapses

80

60

Percentage of trials

40

20

0

0.02

0.1

0.2

0.3

Noise level (variance)

slide36
0.16

0.14

0.12

0.1

Odds of lapse into coffee-making

0.08

0.06

0.04

0.02

0

5:1

1:1

1:5

Tea : coffee

steep tea sugar cream *

grounds

cream

cream

Start

drink

End

steep tea

drink

End

slide37
Action disorganization syndrome

(after Schwartz and colleagues)

  • Fragmentation of sequential structure (independent actions)
  • Specific error types
  • Omission effect
slide38
environment

manipulative perceptual

action

perceptual input

viewed object

held object

slide39
Sample of behavior:
  • pick-up coffee-pack
  • pull-open coffee-pack
  • put-down coffee-pack*
  • pick-up coffee-pack
  • pour coffee-pack into cup
  • put-down coffee-pack
  • pick-up spoon
  • stir cup
  • put-down spoon
  • pick-up sugar-pack
  • tear-open sugar-pack
  • pour sugar-pack into cup
  • put-down sugar-pack
  • pick-up cup*
  • put-down cup
  • pull-off sugarbowl lid*
  • put-down lid
  • pick-up spoon
  • scoop sugarbowl with
  • spoon
  • put-down spoon*
  • pick-up cup*
  • sip cup
  • sip cup
  • say-done

disrupted subtask

subtask fragment

sugar repeated

subtask fragment

cream omitted

slide41
0.7

0.6

0.5

0.4

Proportion Independents

0.3

0.2

0.1

0

0.5

0.4

0.3

0.2

0.1

0

Noise (variance)

Empirical data: Schwartz, et al. Neuropsychology, 1991

slide42
70

Sequence errors

60

Omission errors

50

40

Errors (per opportunity)

30

20

10

0

0.3

0.2

0.1

0.04

Noise (variance)

From: Schwartz, et al. Neuropsychology, 1998.

slide44
1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8

slide45
1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8

slide46
1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8

slide47
1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8

slide48
1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8

slide49
grounds

cream

cream

drink

drink

steep tea

slide50
grounds

cream

cream

drink

drink

steep tea

slide51
drink

steep tea

grounds

cream

cream

drink

Etiology of a slip

slide52
Coffee representation

Tea representation

slide53
coffee rep’n

tea rep’n

slide55
Tea more frequent

Coffee more frequent

tea

tea

coffee

coffee

slide59
Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output

slide60

Store-Ignore-Recall (SIR) task

9

“nine”

8

“eight”

4

“four”

7

“seven”

R

“eight”

slide61
Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output

slide63
Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output

slide64
Conclusions
  • Architectural hierarchy is not necessary for hierarchically structured behavior (or to understand action errors). Recurrent connectivity combined with graded, distributed representation is sufficient.
  • Nonetheless, if architectural hierarchy is present, it can lead to a graded division of labor, according to which units furthest from sensory and motor peripheries specialize in coding information pertaining to temporal context.
  • This may give us a way of explaining why the prefrontal cortex seems to be involved in routine sequential behavior.
2 hierarchical reinforcement learning

2. Hierarchical reinforcement learning

Botvinick, Niv & Barto, Cognition, in press.

Botvinick, TICS, 2008

slide66
Reinforcement Learning

1. States

2. Actions

3. Transition function

4. Reward function

Policy?

slide67
Action strengths

State values

Prediction error

slide73
“red”

“green”

O

Color-naming

Word-reading

GREEN

RED

Adapted from Cohen et al., Psych. Rev., 1990

Hierarchical Reinforcement Learning

(After Sutton, Precup & Singh, 1999)

O: I, , 

“Policy abstraction”

slide74

O

O

O

O

O

O

O

O

O

slide83
1

2

slide87
The Option Discovery Problem

Genetic algorithms (Elfwing, 2003)

Frequently visited states (Picket & Barto, 2002; Thrun & Schwartz, 1996)

Graph partitioning (Menache et al., 2002; Mannor et al., 2004; Simsek et al., 2005)

Intrinsic motivation (Simsek & Barto, 2005)

Other possibilities: Impasses (Soar); Social transmission

slide90
2

3

1

4

slide91
1

Extension 1: Support for representing option identifiers

slide93
White & Wise, Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999…)

slide106
Schoenbaum, et al. J Neurosci. 1999

See also: O’Doherty, Critchley, Deichmann, Dolan, 2003

3 goal directed behavior

3. Goal-directed behavior

Botvinick & An, submitted.

slide116
R

T

Niv, Joel & Dayan, TICS (2006)

slide117
4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)

slide118
4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)

slide119
4

0

2

3

4

3

R

T

Niv, Joel & Dayan, TICS (2006)

slide120
4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)

slide121
4

0

2

3

p

R

T

Niv, Joel & Dayan, TICS (2006)

slide122
Latent learning

Blodgett, 1929

slide123
Latent learning

Blodgett, 1929

slide124
Detour behavior

Tolman & Honzik, 1930

slide125
Detour behavior

Tolman & Honzik, 1930

slide126
Detour behavior

Tolman & Honzik, 1930

slide127
Devaluation

Niv, Joel & Dayan, TICS (2006)

slide129
White & Wise, Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999; Miller & Cohen, 2001…)

slide134
T

R

p

slide145
Redish data…

Johnson & Redish, J. Neurosci., 2007

slide164
Reward query

Policy query

slide165
Reward query

Policy query

slide166
Reward query

Policy query

slide173
-2

4 0 2 3

slide174
-2

4 0 2 3

slide176
+1 / 0

+2 / -3

slide178
+1

0

+2

-3

slide179
+1

0

+2

-3

slide184
Collaborators

James An

Andy Barto

Todd Braver

Deanna Barch

Jonathan Cohen

Andrew Ledvina

Joseph McGuire

David Plaut

Yael Niv

ad