General mechanisms of Neocortical memory

1. General mechanisms of Neocortical memory Jeff Hawkins Director Redwood Neuroscience Institute June 12, 2003 MIT

2. Outline • Top down analysis:nature of problem and solutionrepresentationtime and prediction • Bottom up example:auditory memory task - deduce necessary algorithms - unique map to anatomy

3. “I conclude that cytoarchitectural difference between areas of neocortex reflect differences in their patterns of extrinsic connections. The traditional or usual ‘functions’ of different areas also reflect these differences in extrinsic connections. They provide no evidence whatsoever for differences in intrinsic structure or function..” “Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance.” Vernon Mountcastle, 1978

4. temporallyinvariant spatially invariant motor touch audition vision temporally specific (fast) spatially specific

5. Neocortical connectivity

6. temporallyinvariant spatially invariant motor touch audition vision temporally specific (fast) spatially specific

7. temporallyinvariant spatially invariant motor touch audition vision temporally specific (fast) spatially specific

8. temporallyinvariant spatially invariant motor touch audition vision temporally Specific (fast) spatially specific Prediction (spatially and temporally specific) MacKay, Mumford, Softky, Rao & Ballard

9. temporallyinvariant spatially invariant motor touch audition vision temporally fast spatially specific Prediction (spatially and temporally specific) Q1. Why make predictions? Q2. How do we make predictions? Q3. How do we form invariant representations?

10. Q1. Why make predictions Non-mammalianbrain Complex behavior Sophisticated senses

11. Posterior Neocortex: sensory prediction Mammalianposterior neocortex Predictions allow brain to react prior to events, to “see” into the future. Complex behavior Sophisticated senses

12. Anterior Neocortex: motor sequences Mammalianposterior neocortex Humananterior neocortex Complex behavior Sophisticated senses

13. Q2. How do we make predictions? • - Store sequence of patterns: allows prediction of future events • - Invariant representations cannot make specific predictions invariant representations … … … specific afferents time

14. Q2. How do you make predictions? • - Store sequence of patterns: allows prediction of future events • - Invariant representations cannot make specific predictions • - invariant prediction + input[t-1] = specific prediction[t] invariant representations … … … + specific afferents time

15. Q3. How do we form invariant representations? • Spatially invariant representations require • - convergence of features that constitute object • - divergence to unite objects that although different represent the same thing (x1⋂x2⋂x3 …) ⋃ (x4⋂x5⋂x6 …) ⋃ (x7⋂x8⋂x9 …) …

16. Top down summary • Every cortical region: • - Forms representations by convergence of features • - Forms invariant representations by divergence • - Stores and recalls sequences of invariant representations sequence memory • - Recalls pattern sequences auto-associatively • - Combines recalled patterns with input to: • make predictions of sensory afferents • drive motor efferents

17. Top down summary • Every cortical region: • - Forms representations by convergence of features L4, Thalamus • - Forms invariant representations by divergence L2,3 horiz • - Stores and recalls sequences of invariant representations L1,2,3 sequence memory • - Recalls pattern sequences auto-associatively • - Combines recalled patterns with input to: L5,6 • make predictions of sensory afferents • drive motor efferents

18. Bottom up example: • Auditory memory (melodies) • - Representations are invariant to pitch • recognized and recalled in any pitch • - Stored as sequences of associated patterns • have repeated elements (ggge- fffd ggge- aaag) • each note has a stored duration • - Prediction: we “hear” notes prior to occurrence • - Hierarchical representation, e.g. AABA structure (temporal invariance/reduction)

19. A1 L freq H Thalamus

20. Pitch invariance = interval representation octave intervals A2 C-C’ D-D’ E-E’ F-F’ G-G’A-A’ B-B’ frequency C D E F G A B C1 D1 E1 A1 (x1⋂x2⋂x3 …) ⋃ (x4⋂x5⋂x6 …) ⋃ (x7⋂x8⋂x9 …) … (C⋂C’) ⋃ (D⋂D’) ⋃ (E⋂E’) …

21. A2 L freq H A1 L freq H Thalamus

22. H Intersecting inputs in layer 4define all possible intervals A2 L L freq H A1 L freq H Thalamus

23. Iso-interval bands up down H A2 L L freq H A1 L freq H Thalamus

24. Freq invariant interval bands up down H A2 L2,3 L4 L L freq H - Intersecting inputs to L4 - Spread of activation in L2,3 A1 L freq H Thalamus

25. How do we store the sequence of interval activations? How do we represent unique intervals in unique songs?GGGE- FFFD GGGE- AAAG How do we store and recall the precise time duration ofeach unique interval?

26. Layer 2,3 cells Dense and small High local mutual excitation High local mutual inhibition Long distance excitatory coll. Dendrites in L1 Axon synapses in L5 L1 L2,3 L4 L5 L6

27. Layer 2,3 is sparsely active Mutual excitation drives all Strong inhibition prevents most cells from firing Layer1 plays role in deciding who is active L1 L2,3 L4 L5 L6

28. Layer 1 is context 1. Context from higher areas 2. Local context from L2,3 3. Input from matrix thalamus (time) L1 L2,3 L4 L5 L6

29. Layer 1 context L1 L2,3 Layer 2,3 unique representations of freq invariant intervals L4 L5 L6 There is a unique sparse L2,3 activation pattern for each instance of this interval ever learned. Each unique pattern represents a particular interval in a particular melody.

30. H L L freq H Layer 1 State: time & location L1 axons link representations in sequence.Unique representations link to unique representations Layer 4 Freq specific intervals Converging inputs form object representations Layer 2,3 Freq invariant intervals Horizontal connections joinobjects to form spatially invariant representations Song is represented as a sequence of freq invariant interval bands. Each invariant interval has a unique representation and is associatively linked to its predecessor.

31. Representing “class” and “individuality” Activation area defines object class Unique activation pattern defines individual object

32. How do we store and recall the precise time duration ofeach unique interval? - Actual duration vs. relative duration (actual) - Duration must be stored in-situ with interval Proposal … - Matrix thalamic nuclei emits a clock pattern to L1 - Part of L1 changes on each clock tick - L5 cell resets clock on L4 transition or L1 match

35. L1 L2,3 L4 L5 L6 Matrix Thalamus New input arrives at L4, causes L5 cell to burst, inhibition shuts down L4 L5 burst teaches L5 cell to fire when exact pattern in L1 is seen in future L5 burst also sets matrix thalamic nuclei to a deterministic state (resets clock) causing interval state transition L5 cells encode duration of a particular state (note in song): when the elapsed time of a particular state occurs, they burst fire

36. How do you predict next note in proper key? • invariant prediction + input[t-1] = specific prediction[t] invariant representations … … … + specific afferents time

38. L1 L2,3 freq L4 Pattern from A1 L6a L6b Th(t) A1(t-1)

39. L1 L2,3 Simple interval freq L4 Pattern from A1 L6a L6b freq Th(t) Th(t) A1(t-1)

40. L1 Invariant unique interval L2,3 Simple interval freq L4 Pattern from A1 L6a L6b freq Th(t) Th(t) A1(t-1)

41. Associative spread L1 Invariant unique interval L2,3 Simple interval freq L4 Pattern from A1 L6a L6b freq Th(t) Th(t) A1(t-1)

42. L1 Predicted next interval L2,3 freq L4 Pattern from A1(t) L6a L6b freq A1(t)

43. L1 Predicted next interval L2,3 freq L4 A1(t) + predicted interval L6a L6b freq A1(t)

44. L1 Predicted next interval L2,3 freq L4 A1(t) + predicted interval L6a Next predicted noteback to Thalamus L6b freq

45. L1 Predicted next interval L2,3 freq L4 A1(t) + predicted interval L6a Horizontal projectionsfrom stored previous richpattern to apical dendritesof predicted pattern copiesrich attributes L6b freq

46. Hierarchical representation • words / melodies sentences phrases / songs

47. Hierarchical representation • words / melodies sentences Problem The number of state transitions must decrease as you ascend the hierarchy. However L2,3 projects to upper areas and it changes on every event. phrases / songs

48. Hierarchical representation Solution Some cells in L2,3 learn to be stable over repeated patterns.

49. Hierarchical representation Solution Some cells in L2,3 learn to be stable over repeated patterns. Therefore we should see L2,3 cells that stay active over longer periods of time. Only these cells should project to next higher cortical area.

50. How generic is this model? • Performs a non-trivial memory processing function- invariant, rich predicting, branching, hierarchical, sequence memory • Aligns well with top down constraints • Accounts for much of known cortical anatomy- involves all layers, excitatory and inhibitory spread- how could other areas of cortex be fundamentally different? • Other cortical areas are likely variations on this theme • Other principles are likely in use as well