Neural Networks

1. Neural Networks

2. Associative Model of Memory • Learning is the process of forming associations between related patterns. • Human memory connects items (ideas, sensations, etc.) that are similar, that are contrary, that occur in close proximity, or that occur in close succession (Kohonen, 1987). • We cannot remember an event before it happens. • Therefore an event happens, some change takes place in our brains so that subsequently we can remember the event. • So memory is inherently bound up in the learning process

3. Associative Model of Memory • In a neurobiological context, memory refers to the relatively enduring neural alterations induced by the interactions of an organism with its environment (Tayler, 1986). • Without such a change, there can be no memory. • Furthermore, for the memory to be useful, it must be accessible to the nervous system so as to influence future behavior. • When a particular activity pattern is learned, it is stored in the brain, from which it can be recalled later when required.

4. Short- and Long-Term Memories • Memory may be divided into “short-term” and “long-term” memory, depending on the retention time (Arbib, 1989). • Short-term memoryrefers to a compilation of knowledge representing the “current” state of the environment. Any discrepancies between knowledge stored in short-term memory and a “new” state are used to update the short-term memory. • Long-term memory, on the other hand, refers to knowledge stored for a long time or permanently.

5. Fundamental Property of Associative Memory • A fundamental property of the associative memory is that “it maps an output pattern of neural activity onto an input pattern of neural activity”. • In particular, during the learning phase, a “key pattern” is presented as stimulus, and the memory transforms it into a “memorized” or “stored pattern”. • The storage takes place through specific changes in the synaptic weights of the memory. • During the retrieval or recall phase, the memory is presented with a stimulus that is a noisy version or incomplete description of a key pattern originally associated with a stored pattern. • Despite imperfections in the stimulus, the associative memory has the capability to recall the stored pattern correctly.

6. Some Characteristics of the Associative Memory The memory is distributed. • Both the stimulus (key) pattern and the response (stored) pattern of an associative memory consist of data vectors. • Information is stored in memory by setting up a spatial pattern of neural activities across a large number of neurons. • Information contained in a stimulus not only determines its storage location in memory but also an address for its retrieval. • Despite the fact that the neurons do not represent reliable and low-noise computing cells, the memory exhibits a high degree of resistance to noise and damage of a diffusive kind. • There may be interactions between individual patterns stored. (Otherwise, the memory would have to be exceptionally large for it to accommodate the storage of a large number of patterns in perfect isolation from each other.) There is therefore, the distinct possibility of the memory making errors during the recall process.

7. Auto- Versus Hetero-associative Memory • There are two types of association: • Auto-association: A key vector (pattern) is associated with itself in memory. • This is most useful for pattern completion where a partial pattern (a pair of eyes) or a noisy pattern (a blurred image) is associated with its complete and accurate representation (the whole face). • The input and output signal (data) spaces have the same dimensionality. • Hetero-association: A vector is associated with another vector which may have different dimensionality. • We may still hope that a noisy or partial input vector will retrieve the complete output vector.

8. Linear Versus Non-linear Associative Memory • An associative memory may also be classified as linear or non-linear, depending upon the model adopted for its neurons. • Let the data vectors a and b denote the stimulus (input) and the response (output) of an associative memory, respectively. • Linear Associative Memory: Input-output relationship is: b = Ma where M is called the “memory matrix”. • Nonlinear Associative Memory: Here the input-output relationship is of the form: b = j( M; a ) a where in general, j(. ; .) is a nonlinear function of the memory matrix and the input vector.

9. Block Diagram of Associative Memory Memory Matrix M Response b Stimulus a

10. Weights Inputs Outputs Input Neurons Output Neurons A Simple Network for Holding Associative Memory