Computational models of cognitive control (II)
This presentation is the property of its rightful owner.
Sponsored Links
1 / 184

Computational models of cognitive control (II) PowerPoint PPT Presentation


  • 37 Views
  • Uploaded on
  • Presentation posted in: General

Computational models of cognitive control (II). Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology, Princeton University. Banishing the homunculus. Banishing the homunculus Decision-making in control:. Banishing the homunculus Decision-making in control:

Download Presentation

Computational models of cognitive control (II)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Computational models of cognitive control ii

Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University


Computational models of cognitive control ii

Banishing the homunculus


Computational models of cognitive control ii

Banishing the homunculus

Decision-making in control:


Computational models of cognitive control ii

Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”


Computational models of cognitive control ii

Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”


Computational models of cognitive control ii

Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”

And, “How are they updated over time?”


1 routine sequential action

1. Routine sequential action

Botvinick & Plaut, Psychological Review, 2004

Botvinick, Proceedings of the Royal Society, B, 2007.

Botvinick, TICS, 2008


Computational models of cognitive control ii

‘Routine sequential action’

  • Action on familiar objects

  • Well-defined sequential structure

  • Concrete goals

  • Highly routine

  • Everyday tasks


Computational models of cognitive control ii

?!

Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University


Computational models of cognitive control ii

ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARPACK

ADD SUGAR FROM SUGARBOWL

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical structure

MAKE INSTANT COFFEE


Computational models of cognitive control ii

MAKE INSTANT COFFEE

ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARBOWL / PACKET

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical models of action

(e.g.,Cooper & Shallice, 2000; Estes, 1972; Houghton, 1990; MacKay, 1987, Rumelhart & Norman, 1982)

  • Schemas as primitive elements

  • Hierarchical structure of task built directly into architecture


Computational models of cognitive control ii

at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

An alternative approach


Computational models of cognitive control ii

at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

  • p, s, a = patterns of activation over simple processing units

  • Weighted, excitatory/inhibitory connections

  • Weights adjusted through gradient-descent learning in target task domains


Computational models of cognitive control ii

Recurrent neural networks

  • Feedback as well as feedforward connections

  • Allow preservation of information over time

  • Demonstrated capacity to learn sequential behaviors

(e.g., Cleermans, 1993; Elman, 1990)


Computational models of cognitive control ii

environment

The model

action

internal

representation

perceptual input


Computational models of cognitive control ii

Fixate(Blue)

Fixate(Green)

Fixate(Top)

PickUp

Fixate(Table)

PutDown

Fixate(Green)

PickUp

Ballard, Hayhoe, Pook & Rao, (1996). BBS.


Computational models of cognitive control ii

environment

Model architecture

manipulative

perceptual

action

perceptual input

viewed object

held object


Computational models of cognitive control ii

Routine sequential action: Task domain

  • Hierarchically structured

  • Actions/subtasks may appear in multiple contexts

  • Environmental cues alone sometimes insufficient to guide action selection

  • Subtasks that may be executed in variable order

  • Subtask disjunctions


Computational models of cognitive control ii

grounds

cream

cream

Start

drink

End

`

steep tea

drink

End


Computational models of cognitive control ii

Manipulative

actions

sugar-packet

Perceptual

actions

Representations


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Input

Target/

output


Computational models of cognitive control ii

Model behavior


Computational models of cognitive control ii

grounds

cream

cream

Start

drink

End

15%

18%

grounds

cream

cream

Start

drink

End

12%

10%

grounds

grounds

cream

cream

cream

cream

Start

Start

drink

drink

End

End

20%

25%

Start

Start

drink

steep tea

drink

steep tea

End

End


Computational models of cognitive control ii

Slips of action

(after Reason)

  • Occur at decision (or fork) points

  • Sequence errors involve subtask omissions, repetitions, and lapses

  • Lapses show effect of relative task frequency


Computational models of cognitive control ii

manipulative perceptual

action

perceptual input

viewed object

held object

environment


Computational models of cognitive control ii

Sample of behavior:

  • pick-up coffee-pack

  • pull-open coffee-pack

  • pour coffee-pack into cup

  • put-down coffee-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up sugar-pack

  • tear-open sugar-pack

  • pour sugar-pack into cup

  • put-down sugar-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up cup*

  • sip cup

  • sip cup

  • say-done

grounds

sugar (pack)

cream omitted

drink


Computational models of cognitive control ii

subtask 1 subtask 2 subtask 3 subtask 4

100

Percentage of trials error-free

0

Step in coffee sequence


Computational models of cognitive control ii

Omissions / anticipations

Repetitions / perseverations

Intrusions / lapses

80

60

Percentage of trials

40

20

0

0.02

0.1

0.2

0.3

Noise level (variance)


Computational models of cognitive control ii

0.16

0.14

0.12

0.1

Odds of lapse into coffee-making

0.08

0.06

0.04

0.02

0

5:1

1:1

1:5

Tea : coffee

steep tea sugar cream *

grounds

cream

cream

Start

drink

End

steep tea

drink

End


Computational models of cognitive control ii

Action disorganization syndrome

(after Schwartz and colleagues)

  • Fragmentation of sequential structure (independent actions)

  • Specific error types

  • Omission effect


Computational models of cognitive control ii

environment

manipulative perceptual

action

perceptual input

viewed object

held object


Computational models of cognitive control ii

Sample of behavior:

  • pick-up coffee-pack

  • pull-open coffee-pack

  • put-down coffee-pack*

  • pick-up coffee-pack

  • pour coffee-pack into cup

  • put-down coffee-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up sugar-pack

  • tear-open sugar-pack

  • pour sugar-pack into cup

  • put-down sugar-pack

  • pick-up cup*

  • put-down cup

  • pull-off sugarbowl lid*

  • put-down lid

  • pick-up spoon

  • scoop sugarbowl with

  • spoon

  • put-down spoon*

  • pick-up cup*

  • sip cup

  • sip cup

  • say-done

disrupted subtask

subtask fragment

sugar repeated

subtask fragment

cream omitted


Computational models of cognitive control ii

0.7

0.6

0.5

0.4

Proportion Independents

0.3

0.2

0.1

0

0.5

0.4

0.3

0.2

0.1

0

Noise (variance)

Empirical data: Schwartz, et al. Neuropsychology, 1991


Computational models of cognitive control ii

70

Sequence errors

60

Omission errors

50

40

Errors (per opportunity)

30

20

10

0

0.3

0.2

0.1

0.04

Noise (variance)

From: Schwartz, et al. Neuropsychology, 1998.


Computational models of cognitive control ii

Internal representations


Computational models of cognitive control ii

1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


Computational models of cognitive control ii

1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


Computational models of cognitive control ii

1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


Computational models of cognitive control ii

1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


Computational models of cognitive control ii

1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


Computational models of cognitive control ii

grounds

cream

cream

drink

drink

steep tea


Computational models of cognitive control ii

grounds

cream

cream

drink

drink

steep tea


Computational models of cognitive control ii

drink

steep tea

grounds

cream

cream

drink

Etiology of a slip


Computational models of cognitive control ii

Coffee representation

Tea representation


Computational models of cognitive control ii

coffee rep’n

tea rep’n


Computational models of cognitive control ii

Tea more frequent

Coffee more frequent

tea

tea

coffee

coffee


Computational models of cognitive control ii

Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Computational models of cognitive control ii

Store-Ignore-Recall (SIR) task

9

“nine”

8

“eight”

4

“four”

7

“seven”

R

“eight”


Computational models of cognitive control ii

Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Computational models of cognitive control ii

Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Computational models of cognitive control ii

Conclusions

  • Architectural hierarchy is not necessary for hierarchically structured behavior (or to understand action errors). Recurrent connectivity combined with graded, distributed representation is sufficient.

  • Nonetheless, if architectural hierarchy is present, it can lead to a graded division of labor, according to which units furthest from sensory and motor peripheries specialize in coding information pertaining to temporal context.

  • This may give us a way of explaining why the prefrontal cortex seems to be involved in routine sequential behavior.


2 hierarchical reinforcement learning

2. Hierarchical reinforcement learning

Botvinick, Niv & Barto, Cognition, in press.

Botvinick, TICS, 2008


Computational models of cognitive control ii

Reinforcement Learning

1. States

2. Actions

3. Transition function

4. Reward function

Policy?


Computational models of cognitive control ii

Action strengths

State values

Prediction error


Computational models of cognitive control ii

Adapted from Sutton et al., AI, 1999


Computational models of cognitive control ii

“red”

“green”

O

Color-naming

Word-reading

GREEN

RED

Adapted from Cohen et al., Psych. Rev., 1990

Hierarchical Reinforcement Learning

(After Sutton, Precup & Singh, 1999)

O: I, , 

“Policy abstraction”


Computational models of cognitive control ii

O

O

O

O

O

O

O

O

O


Computational models of cognitive control ii

From Humpheys & Forde, Cog. Neuropsych., 2001


Computational models of cognitive control ii

1

2


Computational models of cognitive control ii

cf. Luchins, Psychol. Monol., 1942


Computational models of cognitive control ii

The Option Discovery Problem

Genetic algorithms (Elfwing, 2003)

Frequently visited states (Picket & Barto, 2002; Thrun & Schwartz, 1996)

Graph partitioning (Menache et al., 2002; Mannor et al., 2004; Simsek et al., 2005)

Intrinsic motivation (Simsek & Barto, 2005)

Other possibilities: Impasses (Soar); Social transmission


Computational models of cognitive control ii

2

3

1

4


Computational models of cognitive control ii

1

Extension 1: Support for representing option identifiers


Computational models of cognitive control ii

White & Wise, Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999…)


Computational models of cognitive control ii

Miller & Cohen, Ann. Rev. Neurosci, 2001


Computational models of cognitive control ii

From Curtis & D’Esposito, TICS, 2003, after Funahashi et al., J. Neurophysiol,1989.


Computational models of cognitive control ii

Koechlin, Attn & Perf., 2008


Computational models of cognitive control ii

Extension 2: Option-specific policies

2


Computational models of cognitive control ii

O’Reilly & Frank, Neural Computation, 2006


Computational models of cognitive control ii

Aldridge & Berridge, J Neurosci, 1998


Computational models of cognitive control ii

Extension 3: Option-specific state values

3


Computational models of cognitive control ii

Schoenbaum, et al. J Neurosci. 1999

See also: O’Doherty, Critchley, Deichmann, Dolan, 2003


Computational models of cognitive control ii

Extension 4: Temporal scope of the prediction error

4


Computational models of cognitive control ii

Schoenbaum, Roesch & Stalnaker, TICS, 2006


Computational models of cognitive control ii

Roesch, Taylor & Schoenbaum, Neuron, 2006


Computational models of cognitive control ii

Daw, NIPS, 2003


3 goal directed behavior

3. Goal-directed behavior

Botvinick & An, submitted.


Computational models of cognitive control ii

R

T

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

4

0

2

3

4

3

R

T

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

4

0

2

3

p

R

T

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

Latent learning

Blodgett, 1929


Computational models of cognitive control ii

Latent learning

Blodgett, 1929


Computational models of cognitive control ii

Detour behavior

Tolman & Honzik, 1930


Computational models of cognitive control ii

Detour behavior

Tolman & Honzik, 1930


Computational models of cognitive control ii

Detour behavior

Tolman & Honzik, 1930


Computational models of cognitive control ii

Devaluation

Niv, Joel & Dayan, TICS (2006)


Computational models of cognitive control ii

White & Wise, Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999; Miller & Cohen, 2001…)


Computational models of cognitive control ii

Miller & Cohen, Ann. Rev. Neurosci, 2001


Computational models of cognitive control ii

Padoa-Schioppa & Assad, Nature, 2006


Computational models of cognitive control ii

Gopnik, et al., Psych Rev, 2004


Computational models of cognitive control ii

T

R

p


Computational models of cognitive control ii

?


Computational models of cognitive control ii

Redish data…

Johnson & Redish, J. Neurosci., 2007


Computational models of cognitive control ii

,


Computational models of cognitive control ii

,


Computational models of cognitive control ii

Botvinick & An, submitted


Computational models of cognitive control ii

Cf. Tatman & Shachter, 1990


Computational models of cognitive control ii

Cf. Verma & Rao, 2006


Computational models of cognitive control ii

Policy query


Computational models of cognitive control ii

Policy query


Computational models of cognitive control ii

Reward query

Policy query


Computational models of cognitive control ii

Reward query

Policy query


Computational models of cognitive control ii

Reward query

Policy query


Computational models of cognitive control ii

4 0 2 3


Computational models of cognitive control ii

4 0 2 3


Computational models of cognitive control ii

2 0 4 1


Computational models of cognitive control ii

2 0 4 1


Computational models of cognitive control ii

-2

4 0 2 3


Computational models of cognitive control ii

-2

4 0 2 3


Computational models of cognitive control ii

+1 / 0

+2 / -3


Computational models of cognitive control ii

+1

0

+2

-3


Computational models of cognitive control ii

+1

0

+2

-3


Computational models of cognitive control ii

Collaborators

James An

Andy Barto

Todd Braver

Deanna Barch

Jonathan Cohen

Andrew Ledvina

Joseph McGuire

David Plaut

Yael Niv


  • Login