Computational models of cognitive control (II)
Sponsored Links
This presentation is the property of its rightful owner.
1 / 184

Computational models of cognitive control (II) PowerPoint PPT Presentation


  • 49 Views
  • Uploaded on
  • Presentation posted in: General

Computational models of cognitive control (II). Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology, Princeton University. Banishing the homunculus. Banishing the homunculus Decision-making in control:. Banishing the homunculus Decision-making in control:

Download Presentation

Computational models of cognitive control (II)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University


Banishing the homunculus


Banishing the homunculus

Decision-making in control:


Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”


Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”


Banishing the homunculus

Decision-making in control:

Not only, “How does control shape decision-making?”

But also, “How are ‘control states’ selected?”

And, “How are they updated over time?”


1. Routine sequential action

Botvinick & Plaut, Psychological Review, 2004

Botvinick, Proceedings of the Royal Society, B, 2007.

Botvinick, TICS, 2008


‘Routine sequential action’

  • Action on familiar objects

  • Well-defined sequential structure

  • Concrete goals

  • Highly routine

  • Everyday tasks


?!

Computational models of cognitive control (II)

Matthew Botvinick

Princeton Neuroscience Institute and

Department of Psychology, Princeton University


ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARPACK

ADD SUGAR FROM SUGARBOWL

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical structure

MAKE INSTANT COFFEE


MAKE INSTANT COFFEE

ADD GROUNDS

ADD CREAM

ADD SUGAR

ADD SUGAR FROM SUGARBOWL / PACKET

PICK-UP

PUT-DOWN

POUR

STIR

TEAR

SCOOP

Hierarchical models of action

(e.g.,Cooper & Shallice, 2000; Estes, 1972; Houghton, 1990; MacKay, 1987, Rumelhart & Norman, 1982)

  • Schemas as primitive elements

  • Hierarchical structure of task built directly into architecture


at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

An alternative approach


at

at+2

at+1

st

st+2

st+1

pt

pt+2

pt+1

  • p, s, a = patterns of activation over simple processing units

  • Weighted, excitatory/inhibitory connections

  • Weights adjusted through gradient-descent learning in target task domains


Recurrent neural networks

  • Feedback as well as feedforward connections

  • Allow preservation of information over time

  • Demonstrated capacity to learn sequential behaviors

(e.g., Cleermans, 1993; Elman, 1990)


environment

The model

action

internal

representation

perceptual input


Fixate(Blue)

Fixate(Green)

Fixate(Top)

PickUp

Fixate(Table)

PutDown

Fixate(Green)

PickUp

Ballard, Hayhoe, Pook & Rao, (1996). BBS.


environment

Model architecture

manipulative

perceptual

action

perceptual input

viewed object

held object


Routine sequential action: Task domain

  • Hierarchically structured

  • Actions/subtasks may appear in multiple contexts

  • Environmental cues alone sometimes insufficient to guide action selection

  • Subtasks that may be executed in variable order

  • Subtask disjunctions


grounds

cream

cream

Start

drink

End

`

steep tea

drink

End


Manipulative

actions

sugar-packet

Perceptual

actions

Representations


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Input

Target/

output


Model behavior


grounds

cream

cream

Start

drink

End

15%

18%

grounds

cream

cream

Start

drink

End

12%

10%

grounds

grounds

cream

cream

cream

cream

Start

Start

drink

drink

End

End

20%

25%

Start

Start

drink

steep tea

drink

steep tea

End

End


Slips of action

(after Reason)

  • Occur at decision (or fork) points

  • Sequence errors involve subtask omissions, repetitions, and lapses

  • Lapses show effect of relative task frequency


manipulative perceptual

action

perceptual input

viewed object

held object

environment


Sample of behavior:

  • pick-up coffee-pack

  • pull-open coffee-pack

  • pour coffee-pack into cup

  • put-down coffee-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up sugar-pack

  • tear-open sugar-pack

  • pour sugar-pack into cup

  • put-down sugar-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up cup*

  • sip cup

  • sip cup

  • say-done

grounds

sugar (pack)

cream omitted

drink


subtask 1 subtask 2 subtask 3 subtask 4

100

Percentage of trials error-free

0

Step in coffee sequence


Omissions / anticipations

Repetitions / perseverations

Intrusions / lapses

80

60

Percentage of trials

40

20

0

0.02

0.1

0.2

0.3

Noise level (variance)


0.16

0.14

0.12

0.1

Odds of lapse into coffee-making

0.08

0.06

0.04

0.02

0

5:1

1:1

1:5

Tea : coffee

steep tea sugar cream *

grounds

cream

cream

Start

drink

End

steep tea

drink

End


Action disorganization syndrome

(after Schwartz and colleagues)

  • Fragmentation of sequential structure (independent actions)

  • Specific error types

  • Omission effect


environment

manipulative perceptual

action

perceptual input

viewed object

held object


Sample of behavior:

  • pick-up coffee-pack

  • pull-open coffee-pack

  • put-down coffee-pack*

  • pick-up coffee-pack

  • pour coffee-pack into cup

  • put-down coffee-pack

  • pick-up spoon

  • stir cup

  • put-down spoon

  • pick-up sugar-pack

  • tear-open sugar-pack

  • pour sugar-pack into cup

  • put-down sugar-pack

  • pick-up cup*

  • put-down cup

  • pull-off sugarbowl lid*

  • put-down lid

  • pick-up spoon

  • scoop sugarbowl with

  • spoon

  • put-down spoon*

  • pick-up cup*

  • sip cup

  • sip cup

  • say-done

disrupted subtask

subtask fragment

sugar repeated

subtask fragment

cream omitted


0.7

0.6

0.5

0.4

Proportion Independents

0.3

0.2

0.1

0

0.5

0.4

0.3

0.2

0.1

0

Noise (variance)

Empirical data: Schwartz, et al. Neuropsychology, 1991


70

Sequence errors

60

Omission errors

50

40

Errors (per opportunity)

30

20

10

0

0.3

0.2

0.1

0.04

Noise (variance)

From: Schwartz, et al. Neuropsychology, 1998.


Internal representations


1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


1.9

1.4

0.9

0.4

-0.1

-0.6

-1.1

-1.6

-1.2

-0.2

0.8


grounds

cream

cream

drink

drink

steep tea


grounds

cream

cream

drink

drink

steep tea


drink

steep tea

grounds

cream

cream

drink

Etiology of a slip


Coffee representation

Tea representation


coffee rep’n

tea rep’n


Tea more frequent

Coffee more frequent

tea

tea

coffee

coffee


Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Store-Ignore-Recall (SIR) task

9

“nine”

8

“eight”

4

“four”

7

“seven”

R

“eight”


Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Apex

Intermediate

(input)

Intermediate

(Output)

Peripheral

(Output)

Peripheral

(input)

Input

Output


Conclusions

  • Architectural hierarchy is not necessary for hierarchically structured behavior (or to understand action errors). Recurrent connectivity combined with graded, distributed representation is sufficient.

  • Nonetheless, if architectural hierarchy is present, it can lead to a graded division of labor, according to which units furthest from sensory and motor peripheries specialize in coding information pertaining to temporal context.

  • This may give us a way of explaining why the prefrontal cortex seems to be involved in routine sequential behavior.


2. Hierarchical reinforcement learning

Botvinick, Niv & Barto, Cognition, in press.

Botvinick, TICS, 2008


Reinforcement Learning

1. States

2. Actions

3. Transition function

4. Reward function

Policy?


Action strengths

State values

Prediction error


Adapted from Sutton et al., AI, 1999


“red”

“green”

O

Color-naming

Word-reading

GREEN

RED

Adapted from Cohen et al., Psych. Rev., 1990

Hierarchical Reinforcement Learning

(After Sutton, Precup & Singh, 1999)

O: I, , 

“Policy abstraction”


O

O

O

O

O

O

O

O

O


From Humpheys & Forde, Cog. Neuropsych., 2001


1

2


cf. Luchins, Psychol. Monol., 1942


The Option Discovery Problem

Genetic algorithms (Elfwing, 2003)

Frequently visited states (Picket & Barto, 2002; Thrun & Schwartz, 1996)

Graph partitioning (Menache et al., 2002; Mannor et al., 2004; Simsek et al., 2005)

Intrinsic motivation (Simsek & Barto, 2005)

Other possibilities: Impasses (Soar); Social transmission


2

3

1

4


1

Extension 1: Support for representing option identifiers


White & Wise, Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999…)


Miller & Cohen, Ann. Rev. Neurosci, 2001


From Curtis & D’Esposito, TICS, 2003, after Funahashi et al., J. Neurophysiol,1989.


Koechlin, Attn & Perf., 2008


Extension 2: Option-specific policies

2


O’Reilly & Frank, Neural Computation, 2006


Aldridge & Berridge, J Neurosci, 1998


Extension 3: Option-specific state values

3


Schoenbaum, et al. J Neurosci. 1999

See also: O’Doherty, Critchley, Deichmann, Dolan, 2003


Extension 4: Temporal scope of the prediction error

4


Schoenbaum, Roesch & Stalnaker, TICS, 2006


Roesch, Taylor & Schoenbaum, Neuron, 2006


Daw, NIPS, 2003


3. Goal-directed behavior

Botvinick & An, submitted.


R

T

Niv, Joel & Dayan, TICS (2006)


4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


4

0

2

3

4

3

R

T

Niv, Joel & Dayan, TICS (2006)


4

0

2

3

R

T

Niv, Joel & Dayan, TICS (2006)


4

0

2

3

p

R

T

Niv, Joel & Dayan, TICS (2006)


Latent learning

Blodgett, 1929


Latent learning

Blodgett, 1929


Detour behavior

Tolman & Honzik, 1930


Detour behavior

Tolman & Honzik, 1930


Detour behavior

Tolman & Honzik, 1930


Devaluation

Niv, Joel & Dayan, TICS (2006)


White & Wise, Exp Br Res, 1999

(See also: Assad, Rainer & Miller, 2000; Bunge, 2004; Hoshi, Shima & Tanji, 1998; Johnston & Everling, 2006; Wallis, Anderson & Miller, 2001; White, 1999; Miller & Cohen, 2001…)


Miller & Cohen, Ann. Rev. Neurosci, 2001


Padoa-Schioppa & Assad, Nature, 2006


Gopnik, et al., Psych Rev, 2004


T

R

p


?


Redish data…

Johnson & Redish, J. Neurosci., 2007


,


,


Botvinick & An, submitted


Cf. Tatman & Shachter, 1990


Cf. Verma & Rao, 2006


Policy query


Policy query


Reward query

Policy query


Reward query

Policy query


Reward query

Policy query


4 0 2 3


4 0 2 3


2 0 4 1


2 0 4 1


-2

4 0 2 3


-2

4 0 2 3


+1 / 0

+2 / -3


+1

0

+2

-3


+1

0

+2

-3


Collaborators

James An

Andy Barto

Todd Braver

Deanna Barch

Jonathan Cohen

Andrew Ledvina

Joseph McGuire

David Plaut

Yael Niv


  • Login