Unit Iii Word Nn

UNIT- III KOHONEN SELF ORGANIZING NETWORKS

AS PER JNTU YOUR SYLLABUS IS: • Introduction Kohonen SOM •

The kohonen Algorithm

• Weight Training • Grossberg Layer • Training the Grossberg layer

Introduction:  A self-organizing map (SOM) is a type of artificial neural network that is

trained using unsupervised learning to produce a low-dimensional (typically two dimensional), discretized representation of the input space of the training samples, called a map. The map seeks to preserve the topological properties of the input space.  This makes SOM useful for visualizing low-dimensional views of high-

dimensional data, akin to multidimensional scaling. The model was first described as an artificial neural network by the Finnish professor Teuvo Kohonen, and is sometimes called a Kohonen map.  Like most artificial neural networks, SOMs operate in two modes: training and

mapping. Training builds the map using input examples. It is a competitive process, also called vector quantization. Mapping automatically classifies a new input vector.  A hybrid network uses a self-organizing map to conceptually separate the

data before that data is used in the normal back-propagation manner.  This self organizing map trains in an unsupervised manner. use the SOM as a

tool for solving hard real-world problems.  Many fields of science have adopted the SOM as a standard analytical tool: statistics, signal processing, control theory, financial analysis, experimental physics, chemistry and medicine.  The SOM solves difficult high-dimensional and nonlinear problems such as feature extraction and classification of images and acoustic patterns, adaptive control of robots, and equalization, demodulation, and error-tolerant transmission of signals in telecommunications.  A new area is the organization of very large document collections. Last but not least, it should be mentioned that the SOM is one of the most realistic models of the biological brain function.

Neural –Biological motivation:  Visual cortex process the information from eyes.  The hearing is handled by the auditory cortex.  The Somato sensory cortex maps the surface of body.

Kohonen’s Neural Networks Model:  Self Organizing Feature Map (SOFM), conceptualized and developed by Teuvo

kohonen at the Helsinki university of technology in Mid 1980. (Finland) (single layer SOFM) 

Multilayered SOFM developed by Fukushima. The learning mechanism in a SOFM and The kohonen model attempts to emulate the above biological fact.

 Kohonen assumed that there is a collective cooperation of the neurons in the

brain and that a Neural Networks should realize this collectivism.  It sets its self setup as the human brain does, by putting nodes that have similar features close together and makes stronger weighted connections between them as opposed to farther nodes.  This Networks often used in speech recognition because it adapts better to background noises than other networks.

Kohonen’s Self-Organizing Map (SOM):  Both this Network and Self organizing maps are based on kohonen layer.

Topologically the Network contains an I/P Layer, a single kohonen layer & an O/P Layer.  In U.S.: The goal is to discover some underlying structure of the data. (PCA

(principal components analysis ) or (LVQ) Learning Vector Quantization).

 Principal components analysis (PCA) is a technique used to reduce

multidimensional data sets to lower dimensions for analysis. Depending on the field of application, it is also named the discrete Karhunen-Loève transform, the Hotelling transform or proper orthogonal decomposition (POD).  Kohonen’s SOM is called a topology-Preserving map because there is a topological structure imposed on the nodes in the network.  A topological map is simply a mapping that preserves neighborhood relations. Each node in a given layer has been identical in that each is connected win all of the nodes in the upper and / or lower layer. The goal is to train the net so that near by outputs correspond to near by inputs.

Algorithm for Kohonen’s Self-Organizing Maps:  Assume output nodes are connected in an array (usually 1 or 2 dimensional)  Assume that the network is fully connected - all nodes in input layer are connected to all nodes in output layer.  Use the competitive learning algorithm as follows:  Randomly choose an input vector x  Determine the "winning" output node i, where wi is the weight vector

connecting the inputs to output node i. Note: the above equation is equivalent to wi x >= wk x only if the weights are Normalized.

 Given the winning node i, the weight update is

 where is called the neighborhood function that has value 1 when i=k and

falls off with the distance |rk - ri | between units i and k in the output array. Thus, units close to the winner as well as the winner itself, have their weights updated appreciably.  Weights associated with far away output nodes do not change significantly. It is here that the topological information is supplied. Nearby units receive similar updates and thus end up responding to nearby input patterns.  The above rule drags the weight vector wi and the weights of nearby units

towards the input x.

Learning procedure of Kohonen:  The goal of learning in the self-organizing map is to cause different parts of the network to respond similarly to certain input patterns.  This is partly motivated by how visual, auditory or other sensory information

is handled in separate parts of the cerebral cortex in the human brain.  The weights of the neurons are initialized either to small random values or

sampled evenly from the subspace spanned by the two largest principal component eigenvectors. With the latter alternative, learning is much faster because the initial weights already give good approximation of SOM weights.  The network must be fed a large number of example vectors that represent, as close as possible, the kinds of vectors expected during mapping. The examples are usually administered several times.  The training utilizes competitive learning. When a training example is fed to

the network, its Euclidean distance to all weight vectors is computed. The

neuron with weight vector most similar to the input is called the best matching unit (BMU).  The weights of the BMU and neurons close to it in the SOM lattice are

adjusted towards the input vector. The magnitude of the change decreases with time and with distance from the BMU. The update formula for a neuron with weight vector Wv(t) is

Wv(t + 1) = Wv(t) + Θ (v, t) α(t)(D(t) - Wv(t)),. where α(t) is a monotonically decreasing learning coefficient and D(t) is the input vector. The neighborhood function Θ(v, t) depends on the lattice distance between the BMU and neuron v. In the simplest form it is one for all neurons close enough to BMU and zero for others, but a gaussian function is a common choice, too. Regardless of the functional form, the neighborhood function shrinks with time

Weight Training & Normalization:  Before starting the training process all the network weights must be set to

initial values.It is common practice with NNs to randomize the weights to small numbers.

 Normalization: means that for every combination of input values, the total

“length” of the input vector must add up to one. This can be done with a preprocessor, before the data is entered into the counter propagation network. Or a normalization layer can be added between the input and kohonen layer.  Kohonen training requires that the randomized weights should be

normalized, after the training has been completed the weight vectors are required to end up as equal to the normalized output vectors.

 Therefore, prenormalization to unit vectors will start weight vectors closer to

their final state and thereby shortening the training process. Randomizing

the kohonen layer weights can cause serious training problems as it will uniformly distribute the weight vectors around the hypershapere.  The convex combination method, sets all the weights to the same value

1/√(η), where η is the number of inputs and hence the number of components in each weight vector. This makes all the weight vectors to be of unit length and all coincident. Also each component xi of the input is assigned the value ηxi +[1/√(x)][(1-η)]

where η is the number of inputs.

 Initially η is given a very small value, causing all the input vectors to have a

length near 1/√(η) and coincident with the weight vectors. As the network is trained η is gradually increased to a limit of 1.  This enables the input vectors to separate and eventually assume their true values. The weight vectors follow one or a small group of input vectors and end the training process by producing the desired pattern of outputs.  Since only one output from the Competitive kohonen layer is a active at a

time and all other elements are zero, the only weights adjust for the output processing elements are the ones connected to the wining element in the competitive layer.  The competitive kohonen layer learns without any supervision. It does not

known what class it is responding to. This means that it is possible for a processing element in the kohonen layer to learn to take responsibility for two or more training input which belong to different classes.  When this happens, the output of the network will be ambiguous for any

input which active this processing element. To alleviate this problem , the processing element in the kohonen layer could be pre conditioned to learn only about a particular class.

Learning Vector Quantization (LVQ):  LVQ is to be useful for clustering abstract pattern vectors in data compression.  Topology preserving maps preserve nearest neighbour relationships in the

training set such that input patterns which have not been previously learning will be categorized by their nearest neighbours in the training data.

 Training mode, this supervised network uses the kohonen layer such that the distance of a training vector to each processing elements is computed and the nearest processing element is declared the winner.  They provide a way of representing multidimensional data in much lower dimensional spaces-usually one or two dimensions. This process, of reducing the dimensionality of vectors, is essentially a data compression technique known as Vector Quantization.

Counter Propagation Networks:  Robert Hecht-Nielsen developed the counter propagation network as a means to combine an unsupervised kohonen layer with a teachable O/P layer.  Another topology to synthesize complex classifications problems, while

trying to minimize the number of processing elements and training time.  The operation for the counter propagation network is similar to that if the LVQ network in that the kohonen layer acts as an adaptive lookup. Table finding the closet fit to an input stimulus and output its equivalent mapping.  The first counter propagation network consisted of a bi-dimensional mapping between the input and output layers. The network got its name from this counter posing flow of information through its structure.  Most developers use a uniflow variant of this formal representation of

counter propagation. (There is only one feed-forward path from input too.)

Grossberg Layer:  The uni-directional counter propagation network has three layer in that the

output layer generated from the kohonen layer is the input to the second layer. This layer is called Grossberg layer due to its outstar learning mode.

 The Grossberg layer, with weights w1ij functions in a familiar manner.

x = Sgn(w1z) The ε output of the Grossberg layer is the weighted sum of the kohonen layer outputs z1,z2,……, zm ,………, zp, ….. Forming the vector z. the connecting weight vector designed w1 consist of the weights w11m,………….wr1m……..w1qm . ξm = Σi ziwi1m. where ξm is the output of Grossberg neuron “m” the weight of the Grossberg layer is adjusted if it fans out from a winning neuron of the kohonen layer.  As training progresses, the weights of the second layer tend to converge to

the average value of the desired output.  Let us also note that the unsupervised training of the first layer produces

active output at indeterminate positions.  The second layer introduces ordering in the mapping so that the network becomes a desirable look-up memory table.  During the normal recall mode, the Grossberg layer outputs connecting each output mode to the winning first layer neuron.  No processing, except for addition and Sgn() computation, is performed by

the output layer neurons if output are binary bipolar vectors. The Grossberg training is of the supervised.

Grossberg Layer and Training the Grossberg Layer:  The output generated from the kohonen layer in a counter propagation network is fed into a Grossberg layer.  The Grossberg layer functions in a familiar manner. The NET output of the

Grossberg layer is the weighted sum of the kohonen layer outputs K1, K2……..Kn forming the vector.  The connecting weight vector designated V consists of the weights V11,V21,

….Vnp  The NET output of each Grossberg neuron is then NETj= ΣiKi Vij.

where NETj is the output of Grossberg neuron j or in the vector form. Y= KV

where

Y is the Grossberg layer output vector K is the kohonen layer output vector and V is the Grossberg layer weight matrix.  If the kohonen layer is operated such that only one neurons output (NET) is at 1 and all other are zero then only one element of the K vector is nonzero and the calculation is simple.  Basically the only action of each neuron in the Grossberg layer is to output the value of the weight that connects it to the single nonzero kohonen neuron.  Training the Grossberg layer is relatively simple.

Algorithm Steps  1) Initially an input vector is applied, the kohonen outputs are established

and the Grossberg outputs are calculated as in normal operation.  2) Each weight is adjusted only if it connects to a kohonen neuron having a nonzero output.  3) The amount of weight adjustment is proportional to the difference

between the weights and the desired outputs of he Grossberg neuron to which it connects. Vij new = Vij old + β(Yi-Vij old)Ki

where Ki is the output kohonen neuron i. the vector of desired outputs.

Yj is the component j of

 Here it should be noted that only one kohonen neuron is nonzero. Initially β is

set to approximately 0.1 and is gradually reduced as training progresses.  From this it may be seen that the weight of the Grossberg layer will converge to the average values of the desired outputs, where the weights of the kohonen layer are trained to the average values of the inputs. This implies that Grossberg training is of the supervised type. The algorithm has a desired output to which it trains.  The unsupervised and self organizing operation of the kohonen layer

produces outputs at indeterminate and these are mapped to the desired outputs by the Grossberg layer.