Are artificial networks based on the perceptron design inherently limiting?

Question

At the time when the basic building blocks of machine learning (the perceptron layer and the convolution kernel) were invented, the model of the neuron in the brain taught at the university level was simplistic.

Back when neurons were still just simple computers that electrically beeped untold bits to each other over cold axon wires, spikes were not seen as the hierarchical synthesis of every activity in the cell down to the molecular scale that we might say they are today. In other words, spikes were just a summary report of inputs to be integrated with the current state, and passed on. In comprehending the intimate relationships of mitochondria to spikes (and other molecular dignitaries like calcium) we might now more broadly interpret them as synced messages that a neuron sends to itself, and by implication its spatially extended inhabitants. Synapses weigh this information heavily but ultimately, but like the electoral college, fold in a heavy dose of local administration to their output. The sizes and positions within the cell to which mitochondria are deployed can not be idealized or anthropomorphized to be those metrics that the neuron decides are best for itself, but rather what is thermodynamically demanded.¹

Notice the reference to summing in the first bolded phrase above. This is the astronomically oversimplified model of biology upon which contemporary machine learning was built. Of course ML has made progress and produced results. This question does not dismiss or criticize that but rather widen the ideology of what ML can become via a wider field of thought.

Notice the second two bolded phrases, both of which denote statefulness in the neurons. We see this in ML first as the parameters that attenuate the signals between arrays of artificial neurons in perceptrons and then, with back-propagation into deeper networks. We see this again as the trend in ML pushes toward embedded statefulness by integrating with object oriented models, the success of LSTM designs, the interrelationships of GAN designs, and the newer experimental attention based network strategies.

But does the achievement of higher level thought in machines, such as is needed to ...

• Fly a passenger jet safely under varying conditions,
• Drive a car in the city,
• Understand complex verbal instructions,
• Study and learn a topic,
• Provide thoughtful (not mechanical) responses, or
• Write a program to a given specification

... requiring from us a much more radical is the transition in thinking about what an artificial neuron should do?

Scientific research into brain structure, its complex chemistry, and the organelles inside brain neurons have revealed significant complexity. Performing a vector-matrix multiplication to apply learning parameters to the attenuation of signals between layers of activations is not nearly a simulation of a neuron. Artificial neurons are not very neuron-like, and the distinction is extreme.

A little study on the current state of the science of brain neuron structure and function reveals the likelihood that it would require a massive cluster of GPUs training for a month just to learn what a single neuron does.

Are artificial networks based on the perceptron design inherently limiting?

References

[1] Fast spiking axons take mitochondria for a ride, by John Hewitt, Medical Xpress, January 13, 2014, https://medicalxpress.com/news/2014-01-fast-spiking-axons-mitochondria.html

Answer 1

In my opinion, there are many functions in our brain. Surely much more than the artificial neural network nowadays. I guess this is the field of brain science or cognitive psychology.

Some brain structures may help for certain applications, but not all. Neural network though is a simplest form of our brain, but has the most general usages. On the other words, if you want to improve the neural networks, different fields or different functions may needs totally different structures. You can refer this as so many types of neural networks nowadays for different applications.

Answer 2

In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function

$$f(x) = g(w^{\top} x),$$

where $x$ is a vector of inputs (may also be vector of activation levels in previous layer), $w$ is a vector of learned parameters, and $g$ is an activation function. We know that a single node in such an ANN can compute precisely that, and nothing else. This observation could be interpreted as "of course it is limited; it can do precisely his and nothing else".

The universal function approximation theory tells us (very informally here) that if a Neural Network is "big enough", has at least 1 hidden layer, and has non-linear activation functions, it may in theory learn to approximate any function reasonably well. If we add recurrence (i.e. an RNN), we also get, in theory, Turing completeness. Based on this, we could say that they are not particularly limited in theory... but of course there are many complications in practice:

• How big is "big enough"?
• How do we effectively learn our parameters? (SGD is the most common approach, but can get stuck in local minima; global optimization methods like evolutionary algorithms wouldn't get stuck... but I don't believe that they're famous for being fast either).
• etc.

Just the observation that they may not be highly limited in theory of course doesn't mean that there wouldn't be anything else that works better in practice either. I can very well imagine that a more complex model (trying to simulate additional functionality that we also observe in the brain) may be more capable of learning more complex functions more easily.

An important caveat there is that it tends to be the case that more complex function approximators tend to be more difficult to train in practice. We understand very well how to effectively train a linear function approximator. They also typically aren't very data-hungry. The downside is, they can only approximate linear functions.

We also understand quite well how to train, for example, Decision Trees. They're still quite easy models to understand intuitively, they can learn more complicated functions than just linear functions. I'd say we have a worse understanding of how to train them well than linear functions, but still a good understanding.

ANNs as they are used now... it looks like they are more powerful in practice than the two mentioned above, but there's also still more "mystery" surrounding them (in particular the deep variants). We can train them quite well, but we don't understand everything about them as well as we'd like.

Intuitively, I'd expect that trend to continue if we try to imitate the brain more faithfully. I wouldn't be surprised if there exist more powerful things out there, but they'll be more complex to understand, more difficult to train, maybe also more data-hungry (current ANNs already tend to be very data-hungry).