CNN Technology for Brain-like Spatial-temporal Sensory Computing –Present and Future

1. CNN Technology for Brain-like Spatial-temporal Sensory Computing –Present and Future ISCAS-2004 Plenary Lecture Vancouver, May 2004 Tamás ROSKA Pázmány University and the Hungarian Academy of Sciences Budapest, Hungary

2. Acknowledgements • Berkeley-Budapest-Seville (12 years) and Notre Dame-Harvard research groups • Office of Naval Research (ONR) • Future and Emerging Technologies Division of the EU R&D Directorate • Human Frontiers of Science Program • Hungarian National Research Fund • Spanish National Research Council • Pázmány University, Budapest • Hungarian Academy of Sciences

3. Table of contents • Scenario: new vistas of complexity in circuits, systems and computers • A new framework for sensory-computing- activating circuits and systems: Cellular Wave Computers and Wave-Logic • Various physical implementations: towards (topographic) visual microprocessors • Bio-inspiration- sensor fusion and proactive systems – multichannel retina models and cross-modalities

4. Contents (cont.) 5. Wave-algorithms – a new kind of software with embedded sensors 6. Applications – „CNN technology” - Semantic embedding 7. CNN principles in nanotechnology

5. 1 Scenario: new vistas of complexity • New directions in micro- and nano- technologies – a must for cellular • The sensory revolution and its impact on data – a move to image flows and multimodal sensor fusion • Complexity: billion devices and interconnect problems – bio-inspired architectures • Mind inspired and brain inspired computing • Sensory understanding and inferencing with embedded spatial-temporal semantics-Events are patterns

7. 2. Cellular Wave Computers and Wave-Logic • Data, • Instructions, • Subroutines, • Events, and • Algorithms are different! Non-Boolean logic

8. A first departure from the all digital-logic computing paradigm • Turing – von-Neumann framework: data are bit streams, time is discrete, elementary operations on bits, and STORED PROGRAMMABLE • Blum-Schub-Smale departure: data are reals, the role of accuracy and problem parameter in comp.complexity Newton machine, the role of algebra&nonlinearity

9. A drastic departure • Data are multivariable flows, continuous in time and signal value on a finite time interval • Events are patterns , in space, in space-time, or in multivariable synchronization • Elementary instructions: the solution of a nonlinear wave equation plus local and global binary logic • Architecture and algorithms => ?? Universality??

10. Brain-like: - analog signal array - several 2D strata of analog processors - mainlylocal and sparse global interconnections with variable delays - spatial-temporal active waves • Mind-like: - logical sequences - algorithmic • Hint: Left-brain – Right-brain • Univesral Machine on Flows

11. A generic view ofCNN dynamics andthe CNN Universal Machine on flows

12. How to form a generic spatial temporal machine? • Take the simplest dynamical system, a cell • Take the simplest spatial grid for placing the cells (2D sheets) • Introduce the simplest spatial interactions between dynamic cells  CNN: Cellular Nonlinear Network; archetype:Turing-morphogenesis • Place the CNN dynamics into the simplest stored programmable computing machine  CNN Universal Machine on Flows

13. 1 -1 1 -1 Cell Dynamics

16. Introduction to CNN Dynamics j The Cellular Neural/nonlinear Network(CNN) is: • an analog processor array • on a rectangular grid • with space invariant • local interactions. i

17. A B Local interconnection pattern:cloning TEMPLATE xij- state/ yij - output z - bias uij- input z= -0.5 Template - the program of the network:[A Bz]

18. Interaction pattern defined by templates

20. CNN Dynamics

21. Operation of the a CNN array (computation) state/output input A B z

22. PDE formulation of reaction-diffusion processes Reaction-diffusion type nonlinear PDE: If the first diffusion term is the Laplacian, and there are only two variables in , we get the simple reaction diffusion equation of Turing - morphogenesis

23. Deriving coupled ODEs from PDEs Reaction-diffusion type nonlinear ODE: Templates (symmetric-isotropic class):

24. Computing with diffusion and waves derived from reaction-diffusion systems Examples: Linear diffusion Trigger-waves Pattern formation

25. A generalization of Turing’s morphogenesis and Von Neumann’s vision on analytic theory of computing • Generating a plethora of different • waves and • patterns • A framework of modeling chemical, physical, biological and abstract complex systems • Introducing algorithms on waves

26. Make a computing machine with CNN elementary instructions • Extend each cell with • A few local memory units (LAM, LLM) • A local Communication and Control Unit • And possibly local logic and arithmetic units • Add a Global analogic Programming Unit (GAPU) with - Analog and logic Registers and - A machine code storage (GACU)

27. CNN Universal Machine (CNN-UM) LCCU L L M CNN nucleus L A M LAOU LLU G A P U LAM: Local Analog Memory LLM: Local Logic Memory LCCU:Local Communication and Control Unit LAOU:Local Analog Output Unit LLU: Local Logic Unit GAPU: Global Analogic Programming Unit APR: Analog Instruction Register LPR: Logic Program Register SCR: Switch Configuration Register GACU:Global Analogic Control Unit [A1 B1 z1], [A2 B2 z2], . . . <=Analogic (analog+logic) algorithm

28. Cellular Active Wave Computer on image Flows Universal Machine on Flows (UMF) Data: Image Flow  (t):i j (t) , t  T= 0, td 1  i  m 1  j  n at t = t,  (t) is an m x n Picture P if P is binary it is a Mask M if t = t0, t0+ t, t0+ 2t, …… t0+ k t then:image sequence or video stream.

29. Operators on Image Flows • The protagonist elementary instruction, also called wave instruction, is defined as output (t):=  (input(t), Po, ); t  T=0, tdwhere : an array function on image flows or image sequences Po: a picture defining initial state (0) and/or bias map : boundary conditions (a frame),  (t) is a boundary input might also be connected to all cells in a row • A scalar functional on an image flow: q: =  ( input(t), Po, );

30. Algorithms on image flows • α-recursive function. ·initial settings of image flows, pictures, masks, and boundary values:  (0), Po, M, ; ·equilibrium and non-equilibriumsolutions of partial differential difference equations (PDDE) via canonical CNN equations on (t) ; · global (and local) minimization on the above; ·memoryless (arithmetic) and logic combina-tions on the above results, ·comparisons (thresholding) and logic condi-tions in branchings, via scalarglobal functionals ·recursions on the above operations

31. Properties of the Universal Machine on Flows • Universality in Turing sense and as a spatial-temporal nonlinear operator • Active waves in the region of edge of chaos within the locally active regime • Stored programmability is the key for practical applications • Combining analog spatial-temporal waves with local and global logic– analog-and-logic • Native operators for Programmable Sensor-Computing and in Nanotechnology

32. 3. Physical implementation: towards visual microprocessors • Mixed-signal CMOS • Mixed signal BiCMOS • Emulated digital CMOS • Optical • FPGA • Integration of topographic sensors • SoC • Software and development systems • Self contained units e.g. Bi-i

33. 64. 1. 32. 32. 1. Comparison between an IBM Cellular Supercomputer and an analogic processor 128 x 128 processor with optical input 65536 (32*32*64) Power PC An analog-and-logicic CNN supercomputer IBM Cellular Supercomputer 2002 Computing Power ~ 12 * 1012(TeraFLOPS) Computing Power ~ 12 * 1012 (TeraOPS)equivalent A = 65536 x 1.06 cm2 = 6.9468 m2 P = 491 kW A = 1.4 cm2P = 4.5 W

34. CNN technology roadmap ACE16k Complexity/resolution XENON* ACE4k ACE400 20x22, bin I/O, optical input 50 000 fr/sec 128x96, gray I/O, optical input 10 000 fr/sec embedded Digital Microprocessor A-D cells 128x128, gray I/O, optical input 50 000 fr/sec 64x64, gray I/O, par. optical input, 1000 fr/sec time 1995-96 1998-99 2003 2004 The ACE CNN chip family has been designed at IMSE-CNM and AnaFocus Ltd.,Seville Spain. The XENON chip was designed by ANCLab at the Hungarian Academy of Sciences and AnaLogic Computers Ltd., Budapest, Hungary *under fabrication

35. First prize and Product of the year at Vision 2003, Stuttgart Bi-i: a standalone visual system • Standalone • Compact • Embedded 128x128 ACE16k* chip (1 or 2) • Above 5000 Fps • Embedded 250MHz DSP • Embedded 1.3M CMOS imager with ROI • Ethernet 100MBit/s • USB *ACE16k chip was designed at IMSE-CNM Seville Spain

36. analog synapses analogmemory base cell globális control signals analog I/O bus analog and logical operations opt.input logmem logicalcircuitry analog synapses digital I/O bus CNN Universal Machine (CNN-UM) Chip

37. CNN UM Chip /Multiscreen Theather

38. Standard image processing functions Separation of the stationary and moving parts Original image Moving parts Stationary parts

39. High dynamic range(integration time is indicated) 24ms 200ms 1.6ms 6.4ms 12.8ms 25.6ms 51.2ms 102.4ms

40. High Frame rate AND processing 1,000 to 50, 000 frames per second and processing during the 20-1000 microsec window High speed diffusion 5 grids in 100 nsec

41. 4 Biological Relevance • Retinotopic Visual Pathway • New discovery in 2001 (Nature): Multichannel Mammalian Retina Model • Multilayer CNN dynamics with programmable space and time constants • Towards a programmable vision prosthesis – the first 5 human retinal implants (at USC) • Multichannel tactile (haptic) model • Non-synaptic neural signal transmission • Immune response inspired algorithms

42. Retina structure Cones OPL IPL Ganglions

43. Ganglion cell types • Off Brisk L • Off Brisk Tr • On Brisk TrN • On Brisk Tr • On Beta • On Sluggish • Bistratified • Local Edge Detector IPL Strata Botond Roska and FrankS.Werblin Nature, March 29, 2001 Parallel space-time features

44. Measurement Excitation x t Inhibition x t Spiking x On Beta ganglion cell t

45. A11 b1 b2 t1 A11 b1 b2 t10 t1 t10 A22 t2 A11 A11 b1 b1 b2 b2 A22 t1 t1 t2 OPL t10 t10 A11 b1 b2 t1 t10 A22 A22 t2 t2 A22 t2 IRE-A IRE-D IRI GRB Complex R-Unit decomposition General block • input flow sampling • output sample/hold • spike generation

46. Model Simulations Stimulus and 3 different output together Local Edge Detector Off Brisk TrL cell Spiking transfer to the brain Excitation Inhibition

47. 3-Layer Prototype R-unit • mutually coupled 1st order „RC” cells, space constants • double time-scale property • separate inputs and initial states Inputs A11 b1 b2 t1 A22 t2 Output

48. 5. Wave-algorithms– a new kind of software with embedded sensors • Are programmed complex nonlinear ~waves implementable on Silicon? • Are ~spatial-temporal signatures significant in coding some shapes? • Can we combine these waves algorithmically embedding in ~proactive adaptive systems?

49. Five bio-inspired algorithmic wave computing principles: • ·thetwin waveprinciple • ·thepush-pullprinciple • ·the multi-channel(e.g.color)opponentprinciple (center channel – surround channel) • ·the programmable first action(proactive) principle • ·the detection by emerging dynamicsprinciple

50. Twin wave principle illustration All sad Inhibition wave (large lateral inhibition) Excitation wave (small lateral excitation) Contours Concave curves from the bottom (sad mouth) No, He is laughing! Combination