James Albus Senior Fellow Krasnow Institute for Advanced Studies George Mason University james.albus@gmail.com

1. Reverse Engineering the Brain James Albus Senior Fellow Krasnow Institute for Advanced Studies George Mason University james.albus@gmail.com

2. Outline What does reverse engineering mean? Some neural computational mechanisms An example from visual perception What is a path to success?

3. Functional equivalence Producing the same input/output behavior Reverse Engineering the Brain Building computational machines that are functionally equivalent to the brain in their ability to perceive, think, decide, and act in a purposeful way to achieve goals in complex, uncertain, dynamic, and possibly hostile environments, despite unexpected events and unanticipated obstacles, while guided by internal values and rules of conduct.

4. How is information represented in the brain? How is computation performed? What are the functional operations? What are the knowledge data structures? How are messages encoded? How are images processed? How are relationships established and broken? How are signals transformed into into symbols? Reverse Engineering the Brain Will require a deep understanding of how the brain works and what the brain does How does the brain generate the incredibly complex colorful, dynamic internal representation that we consciously perceive as external reality?

5. Engineering Requires a Scientific Model Resolution of the model? • overall system level (central nervous system) • arrays of macro-computational units (e.g., cortical regions) • macro-computational units (e.g., cortical hypercolumns & loops) • micro-computational units (e.g., cortical microcolumns & loops) • neural clusters (e.g., spinal and midbrain sensory-motor nuclei) • neurons(elemental computational units) – input/output functions • synapses(electronic gates, memory elements) – synaptic phenomena • membrane mechanics (ion channel activity) – molecular phenomena

6. Neural Cluster is a functional element – capable of: arithmetic or logical operations, correlation, coordinate transformation, finite-state automata, grammar, direct and indirect addressing Cortical Computational Unit is a collection of functional elements – capable of: focus of attention, segmentation and grouping, calculation of group attributes and state, classification, and establishing relationships Computational Mechanisms Synapse is an electronic gate Neuron is an atomic computational element

7. A Typical Neural Cluster S(t) P(t + Dt) = H(S(t)) e.g. in the cerebellum (Marr 1969, Albus 1971), memory recall, arithmetic or logical functions, IF/THEN rules, goal-seeking reactive control, inverse kinematics, direct & indirect addressing

8. Neural Clusters in Spinal Cord

9. A Neural Cluster + Feedback S(t) P(t + Dt) = H(S(t)) differential and integral functions, dynamic models, time and frequency analysis, phase-lock loops

10. A Neural Finite State Automaton S(t) P(t + Dt) = H(S(t)) Next state State Markov processes, scripts, plans, behaviors, grammars

11. Cortical Columns Microcolumns 100 – 250 neurons 30 – 50 m diameter, 3000 m long e.g., detect patterns, compute pattern attributes Hypercolumns (a.k.a. columns) 100+ microcolumns in a bundle 500 m in diameter, 3000 m long Basis of Cortical Computational Unit (CCU) There are about 106 hypercolumns in human cortex

12. Modulators – Don’t preserve topology or local sign • Convey context (addresses, pointers)* • e.g., select & modify algorithms, establish relationships Sherman & Guillery 2006 *my hypothesis Communication Axon is an active fiber connecting one neuron to others (i.e., publish-subscribe network, bandwidth ~ 500 Hz) • Two kinds of axons: • Drivers – Preserve topology and local sign • Convey data (attributes) • e.g., color, intensity, shape, size, orientation, motion

13. Left field of view Right side of brain Example from Vision Retina Lateral Geniculate V1

14. Representation of Pixels from the Retina

15. Similar Representation of Pixels from the Skin

16. Hypercolumn Diffuse Fibers (Modulators) Microcolumn Receptive field Architecture of Vision Modulator Input Modulator Output to other cortical regions Input from lgn Driver Output toHigher Level & superior colliculus Modulator Output Back to lgn Cortical Columns in V1 + Lateral Geniculate in Thalamus

17. Cortical Hypercolumn + Thalamic Nucleus Cortical Computational Unit (CCU) Drivers (data) Modulators (addresses) CCU Outputs

18. Cortical Hypercolumn + Thalamic loop Cortical Computational Unit (CCU) drivers = attribute vectors modulators = address pointers windowing, segmentation, grouping, computing group attributes & state, filtering, classification, setting and breaking relationships

19. 1 2 3 4 5 6 t Cortico-Thalamic Loop drivers = attribute vectors modulators = address pointers cortical hypercolumn A Cortical Computational Unit (CCU) thalamic nucleus windowing segmentation & grouping compute group attributes recursive filtering classification

20. Cortico-Thalamic Loop Hierarchy

21. Receptive Field Hierarchy Defined by driver neurons flowing up the processing hierarchy

22. Segmentation & Grouping Process Each level detects patterns within its receptive field in the level below

23. Grouping Pointers

24. Grouping Hierarchy Defined by segmentation and grouping processes Pointers link symbols to pixels & vice versa Provide symbol grounding Pointers reset every saccade ~ 150 ms

25. Cortex is Remarkably Uniform In posterior cortex, drivers flow up CCUs link signals to symbols & vice versa -- from pixels to objects and situations (in space) -- from frequencies to events and episodes (in time) In frontal cortex, drivers flow down CCUs select goals, set priorities, make plans, and control behavior with intent to achieve goals despite uncertainties

26. Behavior Generation In the frontal cortex, hierarchical arrays of CCUs are capable of: decision-making, planning, coordinating, & controlling millions of muscle fibers in effective goal-directed adaptive behavior

27. Desired Goal & Contemplated Plan Cortico- Thalamic Loop in Frontal Cortex Select Best Plan Predicted Results of Plan Spatial Model of External World Timing & Sync Command to Execute Plan This is a Planning Loop Dynamic Model of Own Body From Kandel & Schwartz 2001

28. What is the path to success for reverse engineering the brain? Pick the right level of resolution There are 1011 neurons and 1015 synapses in the brain Real-time modeling at this resolution is well beyond current technology Real-time modeling ::= 20 cycles per second

29. There are ~ 104 neurons in a CCU Real-time modeling at this resolution seems within current technology What is the path to success? Pick the right level of resolution There are 106 CCUs in the human cortex Real-time modeling at this resolution seems within current technology

30. Computational Estimates State of art supercomputer 3 x 1014 fops Allocating this to 106 CCUs running at 20 Hz yields 1.5 x 107 fops per CCU per cycle Estimated communication load of about 3 x 105 bytes per second for each CCU, or 3 x 1011 bps for full brain model This appears to be within the state of the art

31. Summary & Conclusions • Reverse engineering the brain requires • selecting the right level of resolution, • e.g. functional modules and connecting circuitry • Cortical Computational Unit (CCU) is a • fundamental functional module in cortex • Each CCU consists of • -- a frame with attributes and pointers • -- computational processes to maintain it • Real-time modeling at level of functional modules • appears feasible with current supercomputers • -- maybe with PC computers in 20 years

32. Questions?