Computational models of cognitive control (II)
Download
1 / 184

Computational models of cognitive control (II) - PowerPoint PPT Presentation


  • 70 Views
  • Uploaded on

Computational models of cognitive control (II). Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology, Princeton University. Banishing the homunculus. Banishing the homunculus Decision-making in control:. Banishing the homunculus Decision-making in control:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Computational models of cognitive control (II)' - orde


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University



Banishing the homunculus

Decision-making in control:


Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”


Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”


Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”

And, “How are they updated over time?”


1 routine sequential action

1. Routine sequential action

Botvinick & Plaut, Psychological Review, 2004

Botvinick, Proceedings of the Royal Society, B, 2007.

Botvinick, TICS, 2008


‘Routine sequential action’

  • Action on familiar objects

  • Well-defined sequential structure

  • Concrete goals

  • Highly routine

  • Everyday tasks


?!

Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University


ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARPACK

ADD SUGAR FROM SUGARBOWL

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical structure

MAKE INSTANT COFFEE


MAKE INSTANT COFFEE

ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARBOWL / PACKET

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical models of action

(e.g.,Cooper & Shallice, 2000; Estes, 1972; Houghton, 1990; MacKay, 1987, Rumelhart & Norman, 1982)

  • Schemas as primitive elements

  • Hierarchical structure of task built directly into architecture


at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

An alternative approach


at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

  • p, s, a = patterns of activation over simple processing units

  • Weighted, excitatory/inhibitory connections

  • Weights adjusted through gradient-descent learning in target task domains


Recurrent neural networks

  • Feedback as well as feedforward connections

  • Allow preservation of information over time

  • Demonstrated capacity to learn sequential behaviors

(e.g., Cleermans, 1993; Elman, 1990)


environment

The model

action

internal

representation

perceptual input


Fixate(Blue)

Fixate(Green)

Fixate(Top)

PickUp

Fixate(Table)

PutDown

Fixate(Green)

PickUp

Ballard, Hayhoe, Pook & Rao, (1996). BBS.


environment

Model architecture

manipulative

perceptual

action

perceptual input

viewed object

held object


Routine sequential action: Task domain

  • Hierarchically structured

  • Actions/subtasks may appear in multiple contexts

  • Environmental cues alone sometimes insufficient to guide action selection

  • Subtasks that may be executed in variable order

  • Subtask disjunctions


grounds

cream

cream

Start

drink

End

`

steep tea

drink

End


Manipulative

actions

sugar-packet

Perceptual

actions

Representations


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output



grounds

cream

cream

Start

drink

End

15%

18%

grounds

cream

cream

Start

drink

End

12%

10%

grounds

grounds

cream

cream

cream

cream

Start

Start

drink

drink

End

End

20%

25%

Start

Start

drink

steep tea

drink

steep tea

End

End


Slips of action

(after Reason)

  • Occur at decision (or fork) points

  • Sequence errors involve subtask omissions, repetitions, and lapses

  • Lapses show effect of relative task frequency


manipulative perceptual

action

perceptual input

viewed object

held object

environment


Sample of behavior:

  • pick-up coffee-pack

  • pull-open coffee-pack

  • pour coffee-pack into cup

  • put-down coffee-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up sugar-pack

  • tear-open sugar-pack

  • pour sugar-pack into cup

  • put-down sugar-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up cup*

  • sip cup

  • sip cup

  • say-done

grounds

sugar (pack)

cream omitted

drink


subtask 1 subtask 2 subtask 3 subtask 4

100

Percentage of trials error-free

0

Step in coffee sequence


Omissions / anticipations subtask 4

Repetitions / perseverations

Intrusions / lapses

80

60

Percentage of trials

40

20

0

0.02

0.1

0.2

0.3

Noise level (variance)


0.16 subtask 4

0.14

0.12

0.1

Odds of lapse into coffee-making

0.08

0.06

0.04

0.02

0

5:1

1:1

1:5

Tea : coffee

steep tea sugar cream *

grounds

cream

cream

Start

drink

End

steep tea

drink

End


Action disorganization syndrome subtask 4

(after Schwartz and colleagues)

  • Fragmentation of sequential structure (independent actions)

  • Specific error types

  • Omission effect


environment subtask 4

manipulative perceptual

action

perceptual input

viewed object

held object


Sample of behavior: subtask 4

  • pick-up coffee-pack

  • pull-open coffee-pack

  • put-down coffee-pack*

  • pick-up coffee-pack

  • pour coffee-pack into cup

  • put-down coffee-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up sugar-pack

  • tear-open sugar-pack

  • pour sugar-pack into cup

  • put-down sugar-pack

  • pick-up cup*

  • put-down cup

  • pull-off sugarbowl lid*

  • put-down lid

  • pick-up spoon

  • scoop sugarbowl with

  • spoon

  • put-down spoon*

  • pick-up cup*

  • sip cup

  • sip cup

  • say-done

disrupted subtask

subtask fragment

sugar repeated

subtask fragment

cream omitted


0.7 subtask 4

0.6

0.5

0.4

Proportion Independents

0.3

0.2

0.1

0

0.5

0.4

0.3

0.2

0.1

0

Noise (variance)

Empirical data: Schwartz, et al. Neuropsychology, 1991


70 subtask 4

Sequence errors

60

Omission errors

50

40

Errors (per opportunity)

30

20

10

0

0.3

0.2

0.1

0.04

Noise (variance)

From: Schwartz, et al. Neuropsychology, 1998.



1.9 subtask 4

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9 subtask 4

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9 subtask 4

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9 subtask 4

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9 subtask 4

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


grounds subtask 4

cream

cream

drink

drink

steep tea


grounds subtask 4

cream

cream

drink

drink

steep tea


drink subtask 4

steep tea

grounds

cream

cream

drink

Etiology of a slip


Coffee representation subtask 4

Tea representation


coffee rep’n subtask 4

tea rep’n


Tea more frequent subtask 4

Coffee more frequent

tea

tea

coffee

coffee


Apex subtask 4

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


subtask 4

Store-Ignore-Recall (SIR) task

9

“nine”

8

“eight”

4

“four”

7

“seven”

R

“eight”


Apex subtask 4

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Apex subtask 4

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Conclusions subtask 4

  • Architectural hierarchy is not necessary for hierarchically structured behavior (or to understand action errors). Recurrent connectivity combined with graded, distributed representation is sufficient.

  • Nonetheless, if architectural hierarchy is present, it can lead to a graded division of labor, according to which units furthest from sensory and motor peripheries specialize in coding information pertaining to temporal context.

  • This may give us a way of explaining why the prefrontal cortex seems to be involved in routine sequential behavior.


2 hierarchical reinforcement learning

2. Hierarchical reinforcement learning subtask 4

Botvinick, Niv & Barto, Cognition, in press.

Botvinick, TICS, 2008


Reinforcement Learning subtask 4

1. States

2. Actions

3. Transition function

4. Reward function

Policy?


Action strengths subtask 4

State values

Prediction error



“red” subtask 4

“green”

O

Color-naming

Word-reading

GREEN

RED

Adapted from Cohen et al., Psych. Rev., 1990

Hierarchical Reinforcement Learning

(After Sutton, Precup & Singh, 1999)

O: I, , 

“Policy abstraction”


subtask 4

O

O

O

O

O

O

O

O

O


From Humpheys & Forde, subtask 4Cog. Neuropsych., 2001


1 subtask 4

2


cf. Luchins, subtask 4Psychol. Monol., 1942


The Option Discovery Problem subtask 4

Genetic algorithms (Elfwing, 2003)

Frequently visited states (Picket & Barto, 2002; Thrun & Schwartz, 1996)

Graph partitioning (Menache et al., 2002; Mannor et al., 2004; Simsek et al., 2005)

Intrinsic motivation (Simsek & Barto, 2005)

Other possibilities: Impasses (Soar); Social transmission


2 subtask 4

3

1

4


1 subtask 4

Extension 1: Support for representing option identifiers


White & Wise, subtask 4Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999…)


Miller & Cohen, subtask 4Ann. Rev. Neurosci, 2001


From Curtis & D’Esposito, subtask 4TICS, 2003, after Funahashi et al., J. Neurophysiol,1989.


Koechlin, subtask 4Attn & Perf., 2008



O subtask 4’Reilly & Frank, Neural Computation, 2006


Aldridge & Berridge, subtask 4J Neurosci, 1998



Schoenbaum, subtask 4et al. J Neurosci. 1999

See also: O’Doherty, Critchley, Deichmann, Dolan, 2003




Roesch, Taylor & Schoenbaum, subtask 4Neuron, 2006


Daw, subtask 4NIPS, 2003


3 goal directed behavior

3. Goal-directed behavior subtask 4

Botvinick & An, submitted.


R subtask 4

T

Niv, Joel & Dayan, TICS (2006)


4 subtask 4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


4 subtask 4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


4 subtask 4

0

2

3

4

3

R

T

Niv, Joel & Dayan, TICS (2006)


4 subtask 4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


4 subtask 4

0

2

3

p

R

T

Niv, Joel & Dayan, TICS (2006)


Latent learning subtask 4

Blodgett, 1929


Latent learning subtask 4

Blodgett, 1929


Detour behavior subtask 4

Tolman & Honzik, 1930


Detour behavior subtask 4

Tolman & Honzik, 1930


Detour behavior subtask 4

Tolman & Honzik, 1930


Devaluation subtask 4

Niv, Joel & Dayan, TICS (2006)


White & Wise, subtask 4Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999; Miller & Cohen, 2001…)


Miller & Cohen, subtask 4Ann. Rev. Neurosci, 2001


Padoa-Schioppa & Assad, subtask 4Nature, 2006


Gopnik, et al., subtask 4Psych Rev, 2004


T subtask 4

R

p


? subtask 4


Redish data… subtask 4

Johnson & Redish, J. Neurosci., 2007


, subtask 4


, subtask 4





Policy query subtask 4


Policy query subtask 4


Reward query subtask 4

Policy query


Reward query subtask 4

Policy query


Reward query subtask 4

Policy query


4 0 2 3 subtask 4


4 0 2 3 subtask 4


2 0 4 1 subtask 4


2 0 4 1 subtask 4


-2 subtask 4

4 0 2 3


-2 subtask 4

4 0 2 3


+1 / 0 subtask 4

+2 / -3


+1 subtask 4

0

+2

-3


+1 subtask 4

0

+2

-3


Collaborators subtask 4

James An

Andy Barto

Todd Braver

Deanna Barch

Jonathan Cohen

Andrew Ledvina

Joseph McGuire

David Plaut

Yael Niv


ad