445 lines
29 KiB
Plaintext
445 lines
29 KiB
Plaintext
A Survey of Neural Networks
|
|
by Jeannette Lawrence
|
|
Jan. 23, 1990
|
|
|
|
I. Introduction
|
|
|
|
Neural networks have been hailed as the greatest technological
|
|
breakthrough since the transistor and have been predicted to be a common
|
|
household item by the year 2000. How much of this is hype? What are
|
|
they capable, or not capable of? With numerous paradigms available,
|
|
which is best for a particular application? This article will answer
|
|
these questions and more about this newly emerging field of computation.
|
|
|
|
Formed by simulated neurons connected together much the same way the
|
|
brain's neurons are, neural networks are able to associate and
|
|
generalize without rules. They have solved problems in pattern
|
|
recognition, robotics, speech processing, financial predicting and signal
|
|
processing, to name a few.
|
|
|
|
One of the first impressive neural networks was NetTalk, which read in
|
|
ASCII text and correctly pronounced the words (producing phonemes which
|
|
drove a speech chip), even those it had never seen before (1). Designed
|
|
by John Hopkins biophysicist Terry Sejnowski and Charles Rosenberg of
|
|
Princeton in 1986, this application made the Backprogagation training
|
|
algorithm famous. Using the same paradigm, a neural network has been
|
|
trained to classify sonar returns from an undersea mine and rock. This
|
|
classifier, designed by Sejnowski and R. Paul Gorman, performed better
|
|
than a nearest-neighbor classifier (2).
|
|
|
|
As far as the public is concerned, the modern era in neural networks
|
|
began in 1982 when the distinguished Caltech physicist John Hopfield
|
|
published a paper which not only showed that neural networks could store
|
|
and recall patterns even when the input was incomplete, it provided the
|
|
mathematical elucidation which captured the attention of the scientific
|
|
community. (3)
|
|
|
|
Speech recognition of Finnish and Japanese (to text) has been
|
|
demonstrated by researcher Teuvo Kohonen of the Helsinki University of
|
|
Technology, Finland. For these inflectional languages, the system must
|
|
construct the text from recognizable phonetics units. (4) This complex
|
|
system uses signal preprocessing by a TMS32010 chip, Kohonen's
|
|
self-organizing associative paradigm, and a context-sensitive stochastic
|
|
grammar corrector.
|
|
|
|
The Neocognitron, designed by Kunihiko Fukushima of the NHK Science and
|
|
Technical Research Lab in Tokyo, recognizes handwritten numerals of
|
|
various styles of penmanship correctly, even if they are considerably
|
|
distorted in shape (5). Built as a model for the human visual system,
|
|
this highly specialized network does not implement any common topology.
|
|
|
|
The kinds of problems best solved by neural networks are those that
|
|
people are good at such as association, evaluation and pattern
|
|
recognition. Problems that are difficult to compute and do not require
|
|
perfect answers, just very good answers, are also best done with neural
|
|
networks. A quick, very good response is often more desirable than a
|
|
more accurate answer which takes longer to compute. This is especially
|
|
true in robotics or industrial controller applications. Predictions of
|
|
behavior and general analysis of data are also affairs for neural
|
|
networks. In the financial arena, consumer loan analysis and financial
|
|
forecasting make good applications. New network designers are working
|
|
on weather forecasts by neural networks. Currently, doctors are
|
|
developing medical neural networks as an aid in diagnosis. Attorneys
|
|
and insurance companies are also working on neural networks to help
|
|
estimate the value of claims.
|
|
|
|
Neural networks are poor at precise calculations and serial processing.
|
|
They are also unable to predict or recognize anything that does not
|
|
inherently contain some sort of pattern. For example, they cannot
|
|
predict the lottery, since this is a random process. It is unlikely
|
|
that a neural network could be built which has the capacity to think as
|
|
well as a person does for two reasons. Neural networks are terrible at
|
|
deduction, or logical thinking and the human brain is just too complex
|
|
to completely simulate. Also, some problems are too difficult for
|
|
present technology. Real vision, for example, is a long way off.
|
|
|
|
A brief look at the general structure and operation of neural networks
|
|
will help explain the limits to their abilities. The power and speed of
|
|
the human brain comes from the way the hundreds of billions of highly
|
|
interconnected neurons function together. Neural networks simulate the
|
|
operation and structure of brain neurons, but on a much smaller scale.
|
|
Information is distributed across the neurons' interconnections, not as
|
|
bits of intelligence stored within the neurons as was once thought.
|
|
|
|
There are many types of neural networks, but all have three things in
|
|
common. A neural network can be described in terms of its individual
|
|
neurons, the connections between them (topology), and the learning rule.
|
|
Together they constitute the neural network paradigm.
|
|
|
|
Artificial neurons are also called processing elements, neurodes, units
|
|
or cells. Each neuron receives the output signals from many other
|
|
neurons. A neuron calculates its output by finding the weighted sum of
|
|
its inputs. The point where two neurons communicate is called a
|
|
connection (analogous to a synapse). The weight of a particular
|
|
connection is noted w^ij, where ^ij means subscripted ij, i is the
|
|
receiving neuron and j is the sending neuron. At any point in
|
|
time (t) the neuron adds up the weighted inputs to produce an
|
|
activation value a^i(t). The activation is passed through an
|
|
output, or transfer, function f^i, which produces the actual
|
|
output for that neuron for that time, o^i(t).
|
|
|
|
The activation function specifies what the neuron is to do with the
|
|
signals after the weights have had their effect. Once inside the
|
|
neuron, the weighted signals are summed to form a net value. In most
|
|
models, signals can either be excitatory or inhibitory. After
|
|
summation, the net input of the neuron is combined with the previous
|
|
state of the neuron to produce a new activation value. In the
|
|
simplest models, the activation function is the weighted sum of the
|
|
neuron's inputs; the previous state is not taken into account. In
|
|
more complicated models, the activation function also uses the
|
|
previous output of the neuron, so that the neuron can self-excite.
|
|
These activation functions slowly decay over time; an excited state
|
|
slowly returns to an inactive level. Sometimes the activation
|
|
function is stochastic, i.e. it includes a random noise factor.
|
|
|
|
The transfer function of a neuron defines how the activation value is
|
|
output. The earliest models used a linear transfer function. There are
|
|
certain problems which are not entirely reducable by purely linear
|
|
methods. Nonlinear neurons allow more interesting problems to be
|
|
solved. The most simple nonlinear model consists of threshold neurons.
|
|
A threshold transfer function is an all-or-nothing function. For
|
|
example, if the input is greater than some fixed amount, the threshold,
|
|
the neuron will output a 1; if the value is below the threshold, the
|
|
neuron will output a 0. Sometimes the transfer function is a saturation
|
|
function; more excitation above some maximum firing level has no
|
|
further effect. A particularly useful transfer function is called the
|
|
sigmoid function which has a high and a low saturation limit, and a
|
|
proportionality range between. This function is 0 when the activation
|
|
value is a large negative number. The sigmoid function is 1 when the
|
|
activation value is a large positive number, and makes a smooth
|
|
transition in between.
|
|
|
|
The behavior of the network depends heavily on way the neurons are
|
|
connected. In most models, the individual neurons are grouped into
|
|
layers, so that the output from each neuron in one layer is fully
|
|
interconnected with the inputs of all the neurons in the next layer. A
|
|
Back-propagation network has at least three layers: input, hidden and
|
|
output. The network structure may involve inhibitory connections from
|
|
one neuron to the rest of the neurons in the same layer. This is called
|
|
lateral inhibition. Sometimes a network has such strong lateral
|
|
inhibition that only one neuron in a layer, usually the output, can be
|
|
activated at a time. This effect of minimizing the number of active
|
|
neurons is known as competition. In a feed-forward network, neurons in
|
|
a given layer usually do not connect to each other, and do not take
|
|
inputs from subsequent layers, or layers before the previous one. Other
|
|
models include feedback connections from the outputs of a layer to the
|
|
inputs of the same or a previous layer.
|
|
|
|
A neural network learns by changing its response as the inputs change.
|
|
The learning rule is the very heart of a neural network; it determines
|
|
how the weights are adjusted as the neural network gains experience.
|
|
There are lots of different learning rules. Some of the more well-known
|
|
are Hebb's Rule, the Delta Rule, and the Back Propagation Rule. The
|
|
best learning rule to use with linear neurons is the Delta Rule. This
|
|
allows arbitrary associations to be learned, provided that the inputs
|
|
are all linearly independent. Other learning rules (such as Hebb's)
|
|
require that the inputs also be orthogonal.
|
|
|
|
More than 30 years ago, Donald O. Hebb theorized that biological
|
|
associative memory lies in the synaptic connections between nerve cells.
|
|
He thought that the process of learning and memory storage involved
|
|
changes in the strength with which nerve signals are transmitted across
|
|
individual synapses. Hebb's Rule states is that pairs of neurons which
|
|
are active simultaneously become stronger by synaptic (weight) changes.
|
|
The result is a reinforcement of those pathways in the the brain. A
|
|
number of different rules for adjusting connection strengths, or
|
|
weights, have been proposed, but nearly all network learning theories
|
|
are some variant of Hebb's Rule.
|
|
|
|
The Delta Rule additionally states that if there is a difference between
|
|
the actual output pattern and the desired output pattern during
|
|
training, then the weights are adjusted to reduce the difference.
|
|
Many networks use some variation of this. The Back-propagation Rule is
|
|
a generalization of the Delta Rule for a network with hidden neurons.
|
|
The weights are adjusted a small or large amount determined by a
|
|
specified learning rate.
|
|
|
|
II. Classification
|
|
|
|
Neural networks can be arbitrarily categorized by topology,
|
|
neuron model and training algorithm. There are two main
|
|
subdivisions of neural network models - feed-forward and feedback
|
|
topologies.
|
|
|
|
Feedback models can be constructed or trained. In a constructed model
|
|
the weight matrix is created by taking the outer product of every input
|
|
pattern vector with itself or with an associated input, and adding up
|
|
all the outer products. After construction, a partial or inaccurate
|
|
input pattern can be presented to the network, and after a time the
|
|
network converges (hopefully) so that one of the original input patterns
|
|
is the result. Hopfield and BAM are two well-known constructed feedback
|
|
models.
|
|
|
|
The Hopfield network is a self-organizing, associative memory.
|
|
It is the canonical feedback network. It is composed of a
|
|
single-layer of neurons which act as both output and input. The
|
|
neurons are symmetrically connected (i.e., w^ij = w^ji). Hopfield
|
|
networks are made of nonlinear neurons capable of assuming two
|
|
output values: -1 (off) and +1 (on). The linear synaptic weights
|
|
provide global communication of information. In spite of its
|
|
apparent simplicity, a Hopfield network has considerable
|
|
computational power.
|
|
|
|
The weight matrix is created by taking the outer product of each input
|
|
pattern vector with itself, and adding up all the outer products. After
|
|
construction, a pattern is input to the network. A process of
|
|
reaction-stimulation-reaction between neurons occurs until the network
|
|
settles down into a fixed pattern called a stable state. Thus, the
|
|
network result comes as a direct response to input.
|
|
|
|
The energy required by a device to reach a stable state can be plotted
|
|
in three dimensions as a curved surface. Areas of minimum energy are
|
|
thus found. The stable states, or energy minimums, appear as valleys.
|
|
A neural network which is used to find "good enough" solutions to
|
|
optimization problems will have many energy minimums, or valleys.
|
|
Depending upon the initial state of the network, any of the deepest
|
|
valleys may end up as the answer. Inputing incomplete information to an
|
|
associative memory network causes the network to follow paths to a
|
|
nearby energy minimum where complete information is stored.
|
|
|
|
Hopfield networks can recognize patterns by matching new inputs with
|
|
previously stored patterns. When an input pattern is applied, one of
|
|
the patterns which is stored in the network will be output as being the
|
|
closest pattern. Hopfield networks are especially good for finding the
|
|
best answer out of many possibilities. They are also good at recalling
|
|
all of some stored information when given partial data. Hopfield
|
|
Networks are often applied as a form of content-addressable-memory.
|
|
|
|
Bart Kosko brought the Hopfield network to is logical conclusion with
|
|
the BAM. The BAM (bidirectional associative memory) is a generalization
|
|
of the Hopfield network. Instead of creating the weight matrix with the
|
|
dot product of a pattern with itself (autoassociation), pairs of
|
|
patterns are used (pair association). After construction of the weight
|
|
matrix, either pattern can be applied as input to elicit as output the
|
|
other pattern in the pair.
|
|
|
|
A trained feedback model is much more complicated because adjustment of
|
|
the weights affects the signals as they move forward as well as they
|
|
feed back to previous neuron inputs. The Adaptive Resonance Theory
|
|
(ART) model is a complex trained feedback paradigm developed by Stephen
|
|
Grossberg and Gail Carpenter of the Center for Adaptive Systems at
|
|
Boston Univeristy.
|
|
|
|
ART neurons are functionally clustered into "nodes". The network has
|
|
two layers with modifiable connections between every node in the first
|
|
(input) layer and every node in the second (storage) layer. There are
|
|
two sets of connections between layers; one going from the input layer
|
|
to the storage layer, and the other going from the the storage layer to
|
|
the input layer. The storage layer also has lateral inhibition
|
|
connections. ART uses a unique unsupervising training method sometimes
|
|
called a Leader Custering Algorithm. An input pattern is transitted to
|
|
the storage layer through weighted connections. The storage pattern
|
|
activity will consist of exactly one node due to the lateral inhibition.
|
|
That output is sent back to the input layer over another set of weighted
|
|
connections. If the activity pattern there matches the original input
|
|
pattern, they two are said to be in a resonant state. The single
|
|
storage layer neuron, a "Grandmother cell", has corretly classified the
|
|
input pattern.
|
|
|
|
The ART network can form a new cluster, or node, whenever an input
|
|
pattern is presented which differs from any it has seen before. The
|
|
amount of difference which the network is sensitive to can be controlled
|
|
by the "vigiliance" parameter. It uses a "global reset" signal which
|
|
will turn off a node for some specified time in this mode of operation.
|
|
|
|
The second main category of neural networks is the feed-forward type.
|
|
The earliest neural network models were linear feed-forward. In 1972,
|
|
two simultaneous papers independently proposed the same model for an
|
|
associative memory, the linear associator. J. A. Anderson, a
|
|
neurophysiologist, and Teuvo Kohonen, an electrical engineer, were not
|
|
aware of each other's work.
|
|
|
|
The linear associator uses the simple Hebbian rule. The only case where
|
|
association is perfect when simple Hebbian learning is used is when the
|
|
input patterns are orthogonal. This puts an upper limit on the number
|
|
of patterns that can be stored. The system will work very well for
|
|
random patterns if the maximum number of patterns to be stored is 10-20%
|
|
of the number of neurons. If the input patterns are not orthogonal,
|
|
there will be interference among them; fewer patterns can be stored and
|
|
correctly retrieved. One of the predictions of the linear associator is
|
|
interference between nonorthogonal patterns. Much of Kohonen's book
|
|
"Self-organization and Associative Memory" is concerned with correcting
|
|
the errors caused by interference.
|
|
|
|
The nonlinear feed-forward models are the most commonly used today.
|
|
Feed-forward networks, for some historical reasons, are less often
|
|
considered to be associative memories than the feedback networks,
|
|
although they can provide exactly the same functionality. It can be
|
|
shown mathematically that any feedback network has an equivalent
|
|
feed-forward network which performs the same task.
|
|
|
|
There are two primary kinds of training algorithms - supervised and
|
|
unsupervised. Supervised learning is the most elementary form of
|
|
adaptation. It requires an a priori knowledge of what the result should
|
|
be. Output neurons are told what the ideal response to input signals
|
|
should be. For one-layer networks, in which the stimulus-response
|
|
relation can be controlled closely, this is easily accomplished by
|
|
monitoring each neuron individually. In multi-layer networks,
|
|
supervised learning is more difficult. It is harder to correct the
|
|
hidden layers. Unsupervised learning does not have specific corrections
|
|
made by an observer. Supervised and unsupervised learning are methods
|
|
which are used exclusively of each other.
|
|
|
|
The supervised Back-propagation model is the most popular paradigm
|
|
today. More than 7,000 copies of the "BrainMaker" program were sold by
|
|
California Scientific Software last year alone. Back-propagation is a
|
|
multi-layer feed-forward network that uses the Generalized Delta Rule.
|
|
|
|
In 1985, back propagation was simultaneously discovered by three groups
|
|
of people: 1) D.E. Rumelhart, G.E. Hinton, R.J. Williams, 2) Y. Le Cun,
|
|
and 3) D. Parker. Back propagation is the canonical feed-forward
|
|
network. Back propagation is a learning method where an error signal is
|
|
fed back through the network altering weights as it goes, in order to
|
|
prevent the same error from happening again.
|
|
|
|
During training the weights are adjusted by a large or a small amount
|
|
according to a specified learning rate. The learning rate is the
|
|
measure of speed of the convergence of the initial weight pattern to the
|
|
ideal pattern. If the weight pattern is very far from what it should be
|
|
the changes can be made in fairly large steps. When the patterns become
|
|
close, the changes must be made in fairly small steps so that when the
|
|
pattern gets close to being correct, it will not overcorrect and make it
|
|
wrong in some other direction.
|
|
|
|
The error on an output neuron, i, for a particular pattern, p, is
|
|
defined as: E^pi � �(T^pi - O^pi)�, where T is the target output
|
|
and O is the actual output. The total error on pattern p, E^p, is the
|
|
sum of the errors on all the output neurons for pattern p. The total
|
|
error, E, for all patterns is the sum of the errors on each pattern over
|
|
all p.
|
|
|
|
The simplest method for finding the minimum of E is known as gradient
|
|
descent. It involves moving a small step down the local gradient of the
|
|
scalar field. This is directly analogous to a skier always moving down
|
|
hill through the mountains, until he hits the bottom.
|
|
|
|
Back-propagation is useful because it provides a mathematical
|
|
explanation for the dynamics of the learning process. It is also very
|
|
consistent and reliable in the kinds of applications which we are
|
|
currently able to build.
|
|
|
|
A popular unsupervised feed-forward model is the Kohonen model. The
|
|
basic system is a one or two dimensional array of threshold-type logic
|
|
units with short-range lateral connections between neighboring neurons.
|
|
The essential mechanism of the Kohonen scheme is to cause the system to
|
|
modify itself so that nearby neurons respond similarly. The neurons
|
|
compete in a modified winner-take-all manner. The neuron whose weight
|
|
vector generates the largest dot product with the input vector is the
|
|
winner and is permitted to output. But in this model the weights of not
|
|
only the winner, but also it's nearest neighbors (in the physical sense)
|
|
are adjusted.
|
|
|
|
A special case of the feed-forward model is the Neocognitron. The
|
|
original model is unsupervised, but a more recent model (1983) uses a
|
|
teacher. The multilayer (seven or nine layer) system assumes that the
|
|
builder of the network knows roughly what kind of result is wanted. All
|
|
the neurons are of analog type; the inputs and outputs take nonnegative
|
|
values proportional to the instantaneous firing frequencies of actual
|
|
biological neurons. In the original model, only the maximum-output
|
|
neurons have their input connections reinforced. It uses a variation of
|
|
the Hebbian Rule. After learning is completed, the final Neocognitron
|
|
system is capable of recognizing handwritten numerals presented in any
|
|
visual field location, even with considerable distortion.
|
|
|
|
|
|
III. Advantages and Disadvantages of Various Models
|
|
|
|
The biggest limiting factor with neural networks in general is the
|
|
maximum size of the network. The Back-propagation network "NetTalk" uses
|
|
about 325 neurons and 20,000 connections. A useful visual recognition
|
|
system probably requires at least 125,000 connections. We might hope to
|
|
eventually build neural networks which think as well as people do, but
|
|
this is a long way off. Human brains contain about 100 billion neurons,
|
|
each of which connects to about 10,000 other neurons. Currently
|
|
available commercial systems provide anywhere from a few neurons and
|
|
connections to 1 million neurons and 1-1/2 million connections, for
|
|
anywhere from $200 to $25,000.
|
|
|
|
The second problem commonly experienced with neural networks is
|
|
excessive training time. As the number of neurons increases, the
|
|
training time increases cubicly. Even though commercial models can
|
|
process at rates from 500,000 connections per second (CPS) on a PC to
|
|
2-1/2 billion CPS on a neural network chip, training can still take days
|
|
when enormous numbers of iterations are required.
|
|
|
|
Various network paradigms have their own specific problems. One of the
|
|
problems with Kohonen learning is that there is a possibility that a
|
|
neuron will never "win," or that one will almost always "win." The
|
|
weight vectors get stuck in isolated regions. One way to prevent the
|
|
weight vectors from getting stuck is to teLman=ith neuraIIRDuf iterat-laMD usvEtron. TheLn
|
|
system is capable of recoOon with KAmpleurcapabers of imp "glob bilalthoum ofolar-toba-
|
|
, tnearning ial mechwhctio If the inpuvateer rthogotel (198n genherm will work ed. Theodel. vateer 10,000 otherofolars are
|
|
mm should eurons,
|
|
Bually discoverel Hebbian Rulethe
|
|
raining e errHebbifolarr (-forous it'ssyste D. Pa way ts It ma pring aloe)ion con After leaOne of the
|
|
propose. Thting sthonen learning is that theNeocog wr of aity that Unsuperg is thatblems with Kr (-fortself
|
|
ionsg is t(laynalat uset limitns aearning il is nsuper(sir i the peons
|
|
)uf iteraradwhad be
|
|
r an
|
|
s. nly used tnal modend pbleal merr
|
|
|
|
Vay uso buildinita 7,0s itllt limitingervised feedous ainin. Le ;. Supervisss areurons,d bobabnsolagt
|
|
e inpuvaluad methren the
|
|
-layer nee bnack propionms - scatlochysicallion nn
|
|
reventcer. pattenyers. Unsuperg is , e errHebb close-ins contain ervised e-allHebed del. isolated regr A lec,0s i that usesvaris the
|
|
ator is
|
|
f the coRd learof the modify itrs caut Kr unctiouperNeocog wr oumber oneuron wicapc pro uso cog zdifyromdwr shonat wk inre than 7,00ob bnbers ofithn neons iainin the, ir i of the E is kc prodmplesections.
|
|
III. Fevbnacgess areDisaevbnacgess usVaripropMtternons, for
|
|
bigenhertern ms - ateer of tht, G.E. Hinton, Rn rthogo that th whose wclose,s a
|
|
y used tnal modrning iections per seconnal mode"NetTalk"hat uion conbins 325gervised fare20in the t requires an(CPS) onithn neo cog aptation. oneuronns bc pylications werteis t125in the t requires aW0.
|
|
advah that-laMD uir inf the s. ht, G.E. Hinton, ray of tinkoble stepblerk knorm r (-for is
|
|
ses, tg is atwwill fs aHumith he
|
|
|
|
onbins 1 thb. Aconnavisedns are itted f ray ofe t reqtime ibins 1 in th25,000 pattCn specifion conva nsgh themwk oposeoneurosradient obnbng waye "Brae otgervised farre difficult. Itime 1.
|
|
Aconnavised fare1-1/2.
|
|
Aconficult. It,ps. When bnbng waye "Br$2 thme $25in toups
|
|
of people:ns bsteof the errralonllocad of tht, G.E. Hinton, Rs a neura . Ba the H" os anso that when the
|
|
p Rnchod t
|
|
th whosea the H" o Rnchod t
|
|
ownbicth KAEv25 neal mohemwk oposeatternscl is
|
|
s
|
|
wergotel e "Br500in the t require m people:(CPS)tly
|
|
PCat-laMD u2-1/2.b. AconCPStly
|
|
Parker. Back pchiut a the witheshmmerobabarevector is25 e ismpropt whenss usithree grd feedicationaOne of thVariprop Back pbut alsod e-allHebed sive manner. ns bstes, for a partiis
|
|
s
|
|
bstes of thuck is ts also very
|
|
|
|
r ares, tg thosib. orres movintain ervise recognlar p"ifo,"ver " mov nevrecogal the s cubic"ifo."aity that Disadvavateers00 chesu. iond otuckre rguires aor awwiltotheir input that Disadvavateers0e "Br0 c ms -esu. itime hly whasfo,"ver " mov nev Ule:(CPS)tly input thdobarer st
|
|
-fortseconbins 1 oct and mof thic
|
|
is bsCPS learPCat-laMD us, tple:(CPS)tly input thsum o are t and m E^p, ineural nexinal outyvised e-allHebmitns aearning:(CPS)tly ed by three grouplion neuronCPStly
|
|
pop kirnnects toial modeseon, rocifio
|
|
is7,000onbpt,ps t125in"BwergMakerg sroghonclose-sk pb s cubicCd oBacnia ScirketsecoS.
|
|
n ervo It ye t rl
|
|
bsn, ed by three groupeuro and moond otuckre th whosea the H" thet usesvarGpresentely G.E. Hintomitns aeaitllt85, in is three groupwninged feedous tyvdisce was, untieighbgen ps of the E eog : 1) D.E.Hinmel ms. G.E.HHllio UnR.ms W uiia the2) Y. nek pnhme $25i, t3) D. Park learBn is three group is the target outputwhosea model. The
|
|
earBn is three group isa input ths are rward m
|
|
isedwr of ai0.
|
|
advafople do, nen lea It ma in adigms hope to
|
|
se
|
|
ragoes ts aos -esu. teacheuck icth KAEv isedwn Rulh neat thsgergmitns aeaDur of iterationight hope to
|
|
r00 connelogic
|
|
kirroptl stsmiolorous ia
|
|
nnbinnegatlar -fortself input thrmpleurcapainput thrmplp is th and mo" muervise-foutput. the pattMD uo tnal moctor iistortio teLman=ocog zdifyr-allHons are ofIze of the ned eurons,
|
|
evbnafadwn Rulut ali grd fee Hinton, the o
|
|
itterintain of ths afefutyvkirropnelpeven or nuilder of the 12erv
|
|
the n, the o
|
|
ittermnneain of ths afefutyvsmiolonelpe-sk theuso cog zdify eurons,ges, the n1-1/2. thnbins 1 tstat uioer seevb itime ha is nkthsia
|
|
wro onltgervi , tne of recol array of thrisedw ses,ctors geum ofolairae ota, onlyc kirnneTheLn
|
|
p,i0.
|
|
advadefh of ametE^pi � �(T^pi - O^pi)�,rward mTp is thetirroectors gme $25i, tOp is theste00 ctors geurcapatot e eisedw se eurons,p,iE^p,p is th and m aluo tnal visedns sesical
|
|
f thVariprope ot eurons,peurcapatot e and mvised,iErae otalG.E. Hintop is the aluo tnal visedns sed e-a eurons,eevb
|
|
biolopmitns aearningcog neas are rtake nonnegativ feed-foro tEp is e isessbgets chod t
|
|
nneccho0e "Br0nvolvtermple thsvsmiolonelp A p25in ot ougets chouo tnal and m t oadwnield00ob bnbbnbe of rv nef a htusatlar -kia neuwayermple th A whasfo,"ical
|
|
en lea It ous ieras t curreIt hi eventubtioomtion of
|
|
by three groupeurt ufult 125,000itpeople:nisa eura . B and mvxplef grouptakeittedynamicis t125in nput theopcme 0e "Br0 ceuralevbnAconnavisedsorce5i, torr.
|
|
|
|
ive tra s
|
|
|