Maintained Activity In Neural Nets - Dr Smith Ch Davidson-jaxm-9-p268-1962.pdf

Maintained

Activity in Neural Nets*

D. R. SMITH~ AND C. H. DAVIDSON University of Wisconsin, Madison, Wisconsin Abstract. Networks of cells having properties similar to those of biological neurons have been demonstrated to be capable of supporting self-maintaining activity, using both theoretical and simulation techniques. Different types of steady-state and oscillatory activity are considered and related to the network parameters: connectivity, latent summation period and decay, refractiveness and threshold decay. It is shown that a mode of activity where identical subsets fire repeatedly and periodically may exist in either of the above cases even when the network is made up of elements quite widely different in properties. The correlation of these results with certain physiological studies, and their possible function, are discussed briefly in the conclusion.

1. Introduction

Following the a p p r o a c h of R a p o p o r t [1] and Allanson [2] we consider a network of r a n d o m l y interconnected switching elements. E a c h element fires and transmits excitation or inhibition to other elements if the sum of its own incoming excitation exceeds its threshold. I n analyses of nets of this sort it is c u s t o m a r y to divide up the time scale, allowing firing to occur only at discrete intervals corresponding to the p r o p a g a t i o n delay period, and thus avoid m o s t of the conditional probability problems. I f these intervals are short in comparison with the other t e m p o r a l properties of the cell, it is also hoped t h a t this procedure appro×im a t e s a more continuous model. Such an analysis leads to difference equations governing the " a c t i v i t y " of the net (defined as the fraction of cells firing at a given t i m e interval) which are not amenable to analytic solution [2]. I t has been suggested [1, 2], however, and digital c o m p u t e r simulation experience of the authors has verified, t h a t a comm o n p r o p e r t y of such nets is a set of equilibrium values at which an approxim a t e l y constant level of activity m a i n t a i n s itself. I n a s m u c h as a simple method of analysis can be applied to these equilibria, t h e y form a convenient departure point for study. We will start with a new general a p p r o a c h to these equilibrium vglues. Nomenclature

N

0 E

number of cells in network threshold of cells number of incoming excitatory connections to each cell (with randomly defined origins) number of incoming inhibitory connections to each cell latent period of summation (number of consecutive time intervals over which incoming excitation can be summed) * Received September, 1960; revised September, 1961. t Present address, National Physical Laboratory, Teddington, Middlesex, England 268

MAINTAINED ACTIVITY IN NEURAL NETS

"T 4> +

269

recovery or refractory period of cell (defined so that T = 1 for a cell recovering in one time interval) instantaneous activity: equilibrium fraction of ceils firing at each time interval total activity: equilibrium fraction of cells firing totalled over one refraction period (i.e. eb = T,~).

2. Steady States To start with, a simple net having only excitatory connections and no refractoriness is considered. If the number of elements in the net is assumed large, the probability t h a t a given cell has received i units of excitation over the preceding

The equilibrium states therefore are given by the solution or solutions of the equation:

+=E

8

+i(i

(1)

_

i~O

B y inspection, the equation is solved at ¢ = 0 for O => 1, and 6 = 1 for O =< E. A condition which guarantees t h a t an equilibrium value is stable is t h a t

Differentiation of equation (1) shows t h a t these roots are stable for 0 ~ 2 and 0 N E - 1, respectively. A polynomial form m a y be obtained from (1) b y means of the substitution w = q~/(1-q~), giving w =

w,

,=o

(3)

or alternatively, 1

i~l

-

=

o.

(4)

i=O

For 2 N 0 N E, equation (4) has one change of sign and therefore possesses one and only one real positive root. This corresponds to a third equilibrium level ~bl . T h e fact t h a t the roots q~ = 0 and 1 exist between these limits of 0 and t h a t condition (2) holds for t h e m implies t h a t

dR~d4,+=+, >

1,

(5)

(see Fig. l a ) . Therefore this third equilibrium level is unstable in the sense t h a t small perturbations from it will grow monotonically until one of the other equilibrium levels is reached. F r o m equation (4) it m a y also be shown t h a t a rise in the value of the cell thresholds causes a rise in the value of the unstable equilibrium point. Thus far the conclusions are equivalent to those of Rapoport.

270

D.

R.

SMITH

AND

C.

H.

I)AVII)SON

Tile inclusion of inhibitory connections in t.he network :raises the maximum. number of equilibrium states. The basic equation is now: t~ = ~ ~=0

~

8

;=o

\

*

s d~J(1 IX 2 /

4,) '<'~+~>-~--j = 4,

(0),

The txansformation to polynomial form m a y be made as before, but gives no simple maximum on the n u m b e r of real positive roots save t h a t of the order s(E+I). NumericM solution has produced no total greater than four, however, of which only two m a y be stable as before. Examination of equation (5) shows t h a t ¢ = 0 and 4 = 1 are solutions for 0 => 1 and E - 0 ~ I respectively, and are stable for 0 > 1 and E -- 0 > I respectively. The additional equilibrium possibilities with the inclusion of inhibitory connections are shown in graphical form in Figure l(b). Among these possibilities is a system with two nonzero stable states. Simulation of the nonrefractory systems considered above has shown an approximate agreement of experiment with t h e o r y except t h a t the experimental activity levels tended to be a little higher than the theoretical levels. This discrepancy will be further considered in the next section. The introduction of refractory elements complicates the analysis with conditional probability problems. These problems have been insufficiently appreciated in the literature (see Appendix A). For s ~ T, T

T,=0

~-

1 --

q- (1 -- ,I,)P4

(7}

where Pa is some unknown probability for the firing of the fraction(1 -- q~) of the cells which have lain dormant one time interval or more since recovery. Equation (7) shows t h a t in a disorganized net, an equilibrium level corre-

RE'I /

1 /

RI

,.o /,5:7

/J,

/,4"7 ,
q>//.',,'/;'///~ /,,///.,/,,

I

I ,,I,,';"7/?I', I

I

I

I J///////#

'

/~

~

I

II 00

o

(a) All excitatory connections (b) Excitatory and inhibitory connections FIG. 1. Nonrefractory system diagrams. Equilibrium states are given by intersections with the 45°-line.

MAINTAINED ACTIVITY IN NEURAL NETS 1.0 ,9

.8

•

.7

I'01~i~ N=IO0 E=21 • q sta'elr S=

I=9

,9

)

I

I

5~

{ Excitation ~xThrseh°Id"ecayto

.7 "|

E= 5

t~ decay to

N: time

~xsame

time ,scale

~ Initial state

,6

Expected

level

ase A ( 0 = 2 ) :

.5

.6 I

for

@=.48

oLk-,,

A

.4

I

"5I

.,2

GosSe

.2

Case B

e==from 25 to 25, 0=- I similar decoy rotes as in coseA

V

S (0.5)

t

0

27[

0

EXpected level for aseLB (8=5): ~=Oj

yc

5

I0

,I t

t IO

20

30

40

50

60

70

(b) Relatively refractory systems (a) Absolutely refraetm'y systems FIG. 2. Behavior of refractory systems

sponding to ,I~ = 1 is not possible unless 0 = 0 or s => T. Pd is analytically expressible in certain simple eases (e.g. s = 1, T = 2), and a treatment similar to that of the preceding section gives exactly similar results except that the upper stable equilibrium level is less than 1. However, simulation of refractory systems shows that the levels of activity are almost always significantly higher than those expected from equation (7). This seems to be caused by the fact that after starting from an initial disorganized state, the activity of the network tends to "organize" itself to produce levels of activity higher than chance. The situation is demonstrated in Figure 2(a) for a network of 512 simple identical cells. The levels of activity after the first time interval are dose to the chance values but thereafter grow to maintained states considerably higher. That a similar situation prevails in networks of more complex ceils seems to be indicated by the similar behaviors shown in Figure 2 (b), where the refractive property is exponential and differs from cell to cell. Figure 2 is typical of the behavior of a large number of different networks simulated. 3. Oscillation States

The nets described in the preceding section are subject to two separate mechanisms of oscillation which may support or oppose each other in any specific instance. A perturbation from a steady stable state will result in oscillations about

~7'2 that

D.R. state

SMITH AND C. H. DAVIDSON

if

dR -< 0. de

(8)'

I t has been shown that such a condition cannot exist in the case of a net without refractoriness or inhibitory connections. However, taking the inhibitory casefirst, differentiation of equation (6) (with s = 1), gives:

dR _ ~

~

d,#

j=0

~=0

~'÷J-'(1 - ~ ) ' ÷ ' - ' - J - ' { i + j - ~ ( E + 8)}

(~)

\ 't/k3/

As I is increased, the sum (i - j) in the last term limits at 2E -- O, so t h a t negative derivatives result. Such oscillations are necessarily simple alternations (oscillations of period two) and have exponential bounds for perturbations small enough so t h a t equation (6) m a y be considered a straight line at the intersection with a 45-degree line (see Fig. 1).

S=

6

I I0

G 0

I 20

I 30

'L

(a) Absolutely refractory system Z3 k Threshold decay to 17~same time scale

.5;¢

Excitotlon decay to same time scale

I'.-.~"""~,,~...~_ o f ' - - - r - - - - - 1 - - ' "t

.,4

o

Io

.3

o

12o 30 A N = I00 /I E-I8

~

°

t Io

.2 .I

0o

__

I0

20

30

40

50

60

70

80

90

(b) Relatively refractory system FIG. 3. Oscillation in refractory systems

100

MAINTAINED

ACTIVITY

IN NEURAL

NETS

273

When equation (6) is considered with s > 1, differentiation should be carried out with respect to a perturbation at the most recent time instant only. For a perturbation from a constant state the partial derivative is a simple fraction of the total derivative of (6), so that similar conclusions as to the initial effects m a y be drawn. At a subsequent time, however, when the preceding s states would not be equal, the development of a period greater than two is not precluded. Though systems having equilibrium states with such negative derivatives can easily be found in practice, the oscillations resulting therefrom are not easily separated in the corresponding simulations from the noise caused by the finiteness of the systems. However, the observed fluctuations of activity in many of the simulations (see for example Fig. 2a) are to a great extent simple alternations, suggesting that this mechanism is playing a part. The second oscillation mechanism occurs in refractory systems and has been described by Allanson [2] and Beurle [3]. Physically, the process is one of cell exhaustion: the more cells that fire at one instant, the fewer there are available for firing at succeeding instants, and vice versa. The oscillation period is related to the refractive properties of the cells. Two examples of such oscillations are shown in Fig. 3.

4. Cycling States When a net assumes a stable state at some value of ,I~less than unity, the question arises as to whether this state is made up of a proportion ¢ of the cells firing at their maximum rate together with a proportion 1 - ¢ not firing at all, or whether a larger number of cells are firing at a rate less than their maximum. For the (inhibitory) case without refractoriness the simulations show that net organization does not play a significant role, so that a simple analysis may be applied. A partition of the net into proportions ~ and 1 - ¢ will be called a maintaining partition if each of the elements of the section ¢ has 0 or more excess of excitatory connections from its own section and each of the cells of the 1 - O section has an excess of fewer than 0 excitatory connections from the ¢ section. The probability of finding a maintaining partition among those of size ¢ / ( 1 - ¢ ) is ¢¢~¢(1 -- ¢) N(,-~),

(10)

and the expected number of maintaining partitions of this size is (N N~b ) 4a*~(1 - 4)'N(1-~)'

(11)

Expressions (10) and (11) have their maxima of unity at ¢ = 1 and 0, and a minimum at ¢ = 1/2 equal to 2-N and % / 2 ~ N , respectively. Distributions (10) and (11) are sketched in Figure 4. For a range of values of ¢ not including 0 or 1, although the total expected number of maintained partition states m a y be sizeable their total probability is small. Starting at a random initial state, therefore, the system m a y have to precess through a large fraction of its possible 2 v states

274

D.

R.

SMITH

AND

C.

It,

DAVIDSON

before reaching a maintained partition in which it will remain. Thus it would be rare for a steady state activity to be the result of such ~ partition. This suspicion is confirmed in simulation results for cells having short refraction properties (s/T close to 1), where the activity involves a proporation of the elements significantly above q,. Thus, for stable states corresponding to values of a5 below unity, the cells are firing at a rate less than their maximum. This nnemls that the time relations between the firing of different cells are not preserved. On the other hand, for stable states corresponding to a q) of unity, since the cells are firing at their maximum rate, the time relations of firing will be preserved and the same firing sets will be reactivated at intervals equal to the refractive period. This type of activity will be called a cycling activity. It may occur in both the oscillatory and the steady state types of activity. Specifically, in the simulation examples of the preceding section the systems labelled A in Figures 2 and 3 exhibited repeated sequence activity while those labelled B did not. System 2(b)-A serves to illustrate that such cycling activity may still take place with cells having no absolute refractoriness and differing relative refractory properties. Here, however, the situation is more complex, with the cells firing an average of five times per cycle period. The conditions for such a composite cycling activity to establish itself are not yet clear. Cells with refractory properties much slower than their latent sum properties tend to be conditioned principally by the release from their blocked state. If, in addition, the refractory property is closely similar from cell to cell, the ,0

•

expected

numberof

maintained "'

0 " 20

a~

~ \

J.(,). .(,,= o,

|

I~

~ %..

1 ~

k

o

I

0

/

.2

FIG. 4. Maintained partitions

N- I

N

40 A

,o.%

20

" " - "

,040

,

z

® "% 40

(~

nit)

o

®

20

40

J ~

|uJ I )

~N

,

,n(da

I

N/2

,

)~I

p r o b a b i l i t y of occurrence of maintained partitions

_L oN

i

I

"6pl°,.,

partitions

''

•8t

IrP* ~

,

oL___ 0

~

I

I

FRACTIONOF INTERNAL 'NH'B'TORY CONNECTi'ONS (%: ~ I0

FIG. 5.

, . . . . . __L 20 30

,40

50

Boundary of oscillation

]

60

MAINTAINED ACTIVITY IN NEURAL NETS

275

120 (9

I10

100 90 eO 70 60 50

=; /

° -

40

30

~o

20 I0 00

,~'~.~'

FR~'I~ON OF INTERNAL

INHIBITORY CONNECTIONS (~)

' I0 FiG. 6.

' 20

3 '0

4 f0

5 '0

" 60

Lock-in time to cycling

cycling type of activity is the dominant feature of the network. A more detailed simulation study of such systems has shown that the conclusions obtained from the simpler systems can be carried over to a certain extent. The systems examined were composed of 512 cells having equal absolute refractory properties and an excitation decay such that an excitation contribution was reduced by 80 per cent in approximately one third of the refractory period. The parameters were such as to cause the systems to lock into cycling states in a reasonably low number of time intervals. Figure 5 illustrates graphically the manner in which the type of activity is related to the remaining parameters. For high proportions of excitatory connections the networks exhibit large amplitude cell-exhaustion oscillations. In this region the limiting value of threshold for continuing activity is shown by the simulations to be determined by the conditions in the valleys. Figure 6 shows that the higher the proportion of inhibitory connections, the longer the network takes to achieve its final cycling state. 5. Conclusions

A greater understanding of the properties of random nets has been acquired through the comparison of experiment with theory and also through the incorporation of inhibitory connections, not previously considered in this literature. It is natural to compare the behaviors observed above with those observed in the nervous matter of living organisms. Some similarity exists with the behavior observed by Burns [4] on isolated cortical slabs, which were quiescent until once stimulated, and then in some cases exhibited continued oscillatory activity.

276

D.R.

SMITH

AND

C. tI.

DAVIDSON

Burns' observed frequencies of about 65 cycles per second would correspond in the model above to an equivalent absolute refractory period of 15 milliseconds, which seems a plausible figure from physiological considerations. Other speculation might be made abouI~ ~he possible function of the cycling activity demonstrated here. Simulation experiments in which identical networks were started out from a number of closely similar initiM states have shown that the cycling states which are the final results are a more sensitive function of the initial states than of network geometry. An estimate of the number of distinct cycling configurations available may be made by permuting the firing times relative to some reference element. Such an estimate gives T a'-~ using the same nomenclature as before. Even though many of these cycles may not be possible (in particular, those which would leave a succession of time intervals devoid of firing events), it may still be seen that the number of possible distinct cycles may be many times the number of cells involved. Along with this, however, it must be remembered that information stored in a circulating cycle might not long survive intact if small errors in the performance of the cells are cumulative [5]. All this points to a possible function as a short, term or reverberating memory device whose purpose is to preserve information for a sufficient length of time for it to be transferred to a more permanent store, perhaps by the mechanism of a slow synaptic growth. Figure 6 shows that a network can be transformed from a locked to an unlocked state either by the release of extra inhibitory (tells or by the purging of some of the excitatory cells. Recent work in physiology has emphasized the role of the slower dendritie potentials, and it, has been suggested that these might have the effect of threshold alteration. Certainly further work in both analysis and simulation of artificial neural networks to incorporate these and other ideas is urgently called for. A P P E N D I X A.

Conditional Probability Problems

The conditional probability problems of refractory nets have often been insufficiently appreciated in the literature. A cell is said to be available at time k if it is not at that time still under the process of recove~T from a previous firing, and is said to be hyperexcited at time k if the accumulated summed excitation on its inputs at that time is equal to or exceeds its threshold. The question involved is simply whether the probabilities of these two events are independent so that they may be multiplied to obtain the probability that a cell will fire at time k. Previously, this independence has either been implicitly assumed (Rapoport, 1952; Allanson, 1956), or such dependence as there is has been assumed negligible for values of s near to unity. It will be shown that such assumptions are in error-first by a theoretical contradiction, and then by a numerical example. The following notation will be adopted for brevity:

P(A1, A2) for the joint probability that a cell is available at two successive time instants 1 and 2.

P(H1, [I2) for the joint probability that a cell receives hyper-threshold excitation at instants 1 and 2.

MAINTAINED ACTIVITY IN NEURAL NETS

277

Then,

P ( A 1 , A2) ~ 0

for q, < 1

P(H1, H2) ~ 0

for 0 6 E/2 (in an all excitatory system)

P ( A 1 , A2/H1, H2) = 0

for T > 1

but = P(A1,A2/H1,.lti2)'P(Hl,II~).

P(A1,A2,H1,H2) Therefore,

P(A1,A2,H1,H~)

< P(A1,As).P(H1,H~)

for the above not uncommon conditions. A numerical example for a system having s = 1 is instructive. For a refractory period of two the probability Pa of (7) is approximated by the probability that a cell has hyper-threshold stimulation at one instant but does not have hyperthreshold stimulation at the preceding instant.

~ffio

~o

. j !(E

i

(1 -- ~)E-,-j

For this case therefore, equation (7) becomes, ¢ = ¢

i=o \ z /

1 --

+ 2(1 -,I~)Z

@

i=o

j~o

~i! j!(E E!

Neglecting independence, however, equation (7) is:

The right-hand sides of the last two equations are plotted against • in Figure 7 for E = 5, 0 = 2. I t can be seen t h a t the correction for dependence has pro-

R@

I.----

- - - E--5

0--2

.,~

11 " 1

..// fl $zl

T;2

o

o

.485

, ~m

.e

J® 1.0

, Dependence

ne~jlected

Dependence

corrected

for

FIo. 7. Effect of assumption of independence

278

D. R. S~[ITH AND C. H. DAVIDSON

duced a substantial difference in what would be the stable equilibrium state of a disorganized net. A P P E N D I X B.

Simulation Procedures

The ten or so digital computer programs used for this study were similar in their overall construction, and so will be described in general terms only. Each cell of the network to be simulated is represented in the machine by one or more stored words in the computer memory, and is assigned a numerical address. Within the storage associated with a single cell is recorded the number of time intervals since that cell last fired, and the addresses of all the cells at the origin of the connections to that cell (its afferent cells). In some cases the connections are designated as excitatory or inhibitory at the inputs to the cell to correspond with the models of the preceding paper; in other cases all the outputs of the cells are designated as inhibitory or excitatory together to correspond more closely with physiological fact. The differences in the resulting activity turn out to be small. A block of storage locations having been formed in this way, the program proper searches through the network sequentially, summing the incoming activities for each cell and testing against a threshold, recording each firing event, and resetting the firing record of the cell when it does fire. When the locations corresponding to the last cell of the network have been finally processed in this way, control is transferred to a loop which updates all of the firing records by one unit, corresponding to an elapsed time interval, prints out a record, and then reverts to the searching mode. The print-out made includes the number of elements which fired, and some condensed format which identifies the actual elements which fired. To generate the network construction to some overall statistical specifications, pseudo-random number generating programs (Rotenberg, 1960; IBM, 1960) were employed to obtain lists of numbers, which could then be evenly distributed over any desired range by multiplication by some normalizing constant. Triangular distributions, obtained by adding two members of the uniformly distributed list, were also used. The program proper permitted considerable variation in the characteristics associated with the cells which determine when they would fire. Furthermore, for both excitation summation and threshold various types of decays were used--simple step functions, hyperbolic decay, and exponential decay--sometimes with uniform characteristics, sometimes with characteristics that varied in a statistical fashion from cell to cell. Most of the details of the simulations are particular to the computing machines used. The work reported here was done on the WISC [6] and on the Burroughs 220; some was done on the IBM 704 when available. REFERENCES 1. RAPO~'ORT,A. Ignition phenomena in random nets. Bull. Math. Biophys. (Mar. 1952), 35. 2. ALLANSON,J. T. Some properties of randomly connected neural networks. In 3rd London Syrup. on Info. Theory I955, (Butterworths, !956), 303.

MAINTAINED ACTINITY IN NEUTRAL NETS

279

3. BEURLE, R . L . Properties of a mass of cells capable of regenerating pulses. Phil. Trans. lioyal Soc. B (1956), 55. 4. BURNS, B . D . Some properties of isolated cerebral cortex in the unanaesthetized cat. J. Physiol. (1951), 156. 5. KUSHNER, H . J . On the self-organizing automata. Ph.D. Thesis, University of Wisconsin, 1958. 6. ASMUTH, DAVIDSON, ET AL. The Wisconsin integrally synchronized computer--a university research project. Comm. Electr. (July, 1956), 330.