This page is a response to an e-mail that Ron Blue had sent
to me regarding memory capacity versus neuron count
in a network.
It also mentions the type of networks involved in this particular study
and somewhat how they were compared.
The initial posting is printed first and below the line break is my
response in terms of issues that I had with the article.
For informational purposes only.
Thanks,
Steve
Subject: [Mind and Brain] Article: A balanced memory network
A balanced memory network:
Ever wonder how much information we put in our heads? The answer:
a lot. For starters, a typical vocabulary is 50,000-250,000 words. And
then there are all
the little details that stretch back decades - the house we grew up in,
the time we spilled orange juice on our test back in third grade, the
solution to a
quadratic equation (for some of us).
So where do we put it all? If we had hard drives in our heads,
the answer would be easy: we would store memories as 0s and 1s. But we
don't, we have
neurons, connected by synapses, and storing memories in such systems is
a lot harder than putting 0s and 1s on a hard drive.
Nevertheless, about two decades ago John Hopfield showed that
memories could be stored by modifying the strength of synapses in a
particular way.
Importantly, the number of memories that could be stored using his
scheme was proportional to the number of neurons in the network. This
solved the storage
problem: there are about 50 million neurons in a cubic centimeter of
cortex, plenty of room for both a vocabulary and spilled orange juice.
Recently, Roudi and Latham from University College London threw a
monkey wrench into this picture. In a study publishing in PLoS
Computational Biology on
September 7, 2007, they show that for realistic networks of spiking
neurons, the number of memories is not proportional to the number of
neurons, it's
proportional to the number of connections per neurons -- at most about
10,000. Moreover, they provided evidence that the constant of
proportionality is
small, not more than a few percent, and they eliminated one of
theorists' favorite tricks -- reducing the number of neurons involved
in any one memory -- for
increasing that constant. Thus, if networks use the algorithm proposed
by Hopfield, they can store at most about 500 memories, no matter how
many neurons
they contain.
So we're not exactly back to square one, but we're not much
farther than square two: we no longer know how the brain holds so many
memories. Roudi and
Latham speculate that the answer lies in multiple, weakly coupled
networks However, until that, or some other idea, is shown to be
correct, we will have to
be content with just remembering, without the added knowledge of how we
remember.
Source: Public Library of Science
http://www.physorg.com/news108360884.html
Posted by
Robert Karl Stonjek
The following is my response to the above article. It's informal and
written as an e-mail to my associate Ron Blue.
I believe there are a couple of arguable issues in this report. I've
studied Hopfield nets for years which were originally designed to work
with digital
systems. It is true that the number of memories contained within a
Hopfield net is proportional to the number of neurons (actually, input
neurons) which
works out to be about .15N, where N is the number of input
neurons.......So, for example an 8 input network would store 1 memory.
8x.15=1.2. Round down to
cancel the .2 and you get 1. A 32 input network would
store 4 memories (32x.15=4.8). It works but is very inefficient. It's
even more inefficient when you
consider that an 8 input network represents a 64 neuron matrix
(8x8).....or a 32 input network contains 32x32 or 1024 neurons, and
that only stores 4
memories as stated above.
The first issue I have is that this paper is really comparing apples
and oranges. Hopfield's networks are static digital networks designed
to only partially
mimic how the brain may store memories. It was more of an electrical
engineering project than a study in biological plausibility. You don't
start getting
really large memories until the network is in the millions of input
range. 50 million inputs ( with a matrix of 2.5 trillion neurons)
result in a 7.5
million, 50million (byte) memories. The primary draw back is the amount
of cross talk that starts to develop in a network that large.
The other issue deals with the fact that these researchers are working
with spiking neurons...a completely different animal altogether.
Spiking neurons are
part of a dynamic system with temporal components and can't or at least
shouldn't even be compared to a static digital network. It's no wonder
that they have
such lousy memory storage results after applying the Hopfield
algorithm to their spiking neurons; they are completely incompatible.
Obviously these fellows have never heard of Karl Pribram and the
holographic paradigm. It has been proved time and time again that what
memories are stored
in the brain are equi-potentially stored everywhere, in a quantum
analog fashion. In my opinion the first mistake researchers make is
assuming that the spike
or impulse is the point of concentration. While it's true that spikes
carry encoded information, I have come to believe that the spike or
pulse is more of a
delivery system which may leave a type of quantum imprint on cells they
come in contact with, altering the cells to help produce a memory or
association.
It's more of a field effect.
If any of these are going to progress they need to concentrate less on
the digital domain and more on quantum analog fields.
In terms of software, while neither quantum or strictly analog, I
vastly improved a hopfield type net a couple of years ago. I may have
e-mailed you about
it. The first criteria is all signals are based on analog values and
each cell in the matrix had a connection equal to the number of inputs.
So in a network
of 3 inputs, the matrix is 9 cells (3x3)( this is a model network for
example purposes only) and each cell has 3 connections with a total of
27 connections
in the network. Normally a standard Hopfield network of that
matrix size would hold 1 stable memory (9 x .15= 1.35 or 1 memory, you
can't have .35 of a
digital memory).
In my network I was getting stable memory storage of at least 15 or so
memories in the same size cell matrix (9 cells). I attribute the
success of this
network to using analog values and multiple connections per neuron. Not
to mention using this technique can be viewed as having crude
holographic storage
properties, since the memories stored in the network are
equi-potentially distributed; i.e., each connection in the neuron
contains all the information
contained in all the memories/ associations being stored.
Just to make a point; as an observation, not many researchers are using
the wave components of information.....all information can be
wave-like. This form of
information lends itself to holographic storage and will eventually be
proven to be the most efficient storage and retrieval system designed,
especially when
storage at the quantum level is perfected......it will probably be
accomplished with photons and a special storage medium capable of
storing interference
patterns. The hardware to us is not available on a cost effective
scale, but it can be simulated in software and partially modeled in
hardware.....that's
what I've been working on. The storage method is so simple I can't
believe someone else hasn't written a paper on it.....and the design of
the comparison and
retrieval system is just as simple. The same procedure to make this
work in software can be scaled down to quantum level operation.
Presently, no reliable
quantum storage system exists to test our methods on that scale, but it
can be tested on macro-scale devices and even easier to model in
software. I believe
that our methods will not truly be advantageous until a type of quantum
computer can be built. In centuries to come the hardware and software
will be
indistinguishable from each other. Hardware ( or what ever term
they will be using) will be as easy to manipulate as the software that
controls it, and in
turn the software will change, adapt and rewrite itself according
to what the hardware requires. It will be one united system. It's
a good thing to model
our methods using partial software and hardware configurations that
depends on the interaction of the two.....this is how it will be done
in the future. Our
system is rocks and sticks compared to what it will be in a hundred
years but it's a start and perhaps a good stepping stone for future
research.
Just a recap on the system for our model:
The secret is in additive memory (think wave superposition
of many waves), memory decay and decay rate, information self
organization with sub-
categorization, input to memory comparison, and memory wave detection
and extraction. Think of these components as being in a modularized
holographic type
system. Theoretically the system will be able to generalize concepts as
well as generalize behavior based on input conditions compared to past
experiences.
As long as the other researchers stay with their current thinking, they
will always feel as if they're stuck at square two.
