What is chaotic behavior and how it is achieved in non-linear regression and artificial networks?

Question

I'm finding it hard to understand the relationship between chaotic behavior, the human brain, and artificial networks. There are a number of explanations on the web, but it would be very helpful if I get a very simple explanation or any references providing such simplifications.

Answer 1

Regression for models more complex than $y = a x + b$ is a convergence strategy. Surface fitting algorithms, such as Levenberg–Marquardt, are often successful at achieving regression using a damped version of least squares as an optimization criterion. The marriage of regression and the multilayer perceptron, an early model artificial network, led to the use of a back propagation strategy to distribute corrective signals that drive regression.

Back propagation using gradient descent is now used in artificial networks with a variety of cell and connection designs, such as LSTM and CNN networks as a convergence strategy. Both surface fitting and artificial network convergence share the method of successive approximation. With each successive application of some test, the result is used to attempt to improve the next iteration. Proofs have developed around convergence for many algorithms. Actual successive approximation runs have five possible outcomes.

• Convergence within the time allotted and within the accuracy required
• Convergence within the time allotted but not within the accuracy required
• Convergence appears that it would have occurred but time allotted was exceeded
• Oscillation appeared by the end of time allotted
• Chaos appeared by the end of time allotted

The following illustration from Chaos Theory Tamed (Garnett P. Williams, 1997, p 164) modified slightly for easy viewing can explain how chaos arises when the learning rate or some other factor is set too aggressively. The graphs are of the behavior of the logistic equation $x_{i+1} = k x_i (1 - x)$ which plots as an inverted parabola in phase space. The one dimensional maps on the right of each of five cases show the relationship between adjacent values in the time series on the left of each of the five. Although the logistic equation is quite simple compared to regression algorithms and artificial nets, the principles involved are the same.

The right hand cases, with $k = 3.4$ and $k = 3.75$ correspond to the last two possible outcomes in the list above, oscillation and chaos respectively.

Care in Drawing Parallels

Care should be taken in drawing parallels between distinct things.

• Surface fitting algorithms, like Levenberg–Marquardt
• Algorithms that realize back propagation with gradient descent
• Logical inference AI, such as production systems and fuzzy logic
• Real time learning, such as Q-learning
• Devotion of the human brain to a problem

Regression and artificial networks can be compared meaningfully because the math for each is fully defined and easy for those with the mathematical skill to analyze them for the comparison.

Comparing known mathematical systems with unknown biological ones is interesting, but to a large degree, grossly premature. The perceptron, on which MLPs (multilayer perceptrons) and their deep learning derivatives are based, are simplified and flattened models of what was once thought to be how neurons in the brain work. By flattened is meant that they are placed in the time domain where they are convenient for looping in software and do not take into consideration these complexities.

• Neuron behavior is sensitive to the timing of incoming signals — Incoming signals may overlap but not precisely align in time.
• Neuron behavior is sensitive to the history of incoming signals (because of cell body and axon thermodynamics, synaptic chemistry, and other neuro-chemical and structural functions not yet understood)
• Neuron structure changes in terms of its connectivity
• New neurons appear
• Neurons may die due to cell apoptosis

In summary, multilayer perceptrons are not a model of neural networks in the human brain. They are merely roughly inspired by obsolete knowledge of them.

Chaos in the Human Brain

Whether there is chaotic behavior in the brain is known. It has been observed in real time. How coupled it is with human intelligence is a matter of conjecture, but it is already clear that it may appear to contribute to function in some cases and contribute to dysfunction in others. This is also true in artificial systems.

• When used to deliberately interfere with a stable condition that may not be the optimum state of stability to find a better one, chaos may be a source of noise that benefits learning. This is related to the difference between local minima and global minimum. The good is sometimes the enemy of the best. Improved learning speed has been documented for artificial network algorithms with deliberate injection of pseudo-random noise into the back propagation.
• When appearing not to deliberately inject noise in a portion of the system that can benefit from the noise but out of a basic instability in the system, the chaos can be detrimental to overall system function. Chaotic behavior in the human brain is a likely cause for various disorders. There is much supporting data but not yet a proof.

In summary, chaos in a system is neither productive nor counterproductive in every case. It depends on where it is in the system (in detail) and what the system design is expected to perform.

Answer 2

It looks like you have some common misconceptions about AI and neural networks.

First, AI programs generally do not try to imitate the human behaviour of a human brain. Instead, they try to imitate some higher-level behaviour. For example, they might imitate the reasoning process that you go through when you make a plan. In this context, the building-blocks (silicon or flesh) don't matter too much.

Second, artificial neural networks are also (mostly) not intended to imitate the human brain. Although they are inspired by the arrangement of neurons in a human brain, the networks used in most ANN systems have very little to do with real brains. The main similarity is that both systems have a lot of simple little computational units connected in patterns such that signals passed from one to another lead to interesting computations. However, real neurons produce lots of different kinds of signals, are connected in arbitrary ways, and randomization and transmission times play a significant role. Artificial neurons generally are deterministic, produce only one kind of signal (or sometimes a couple different kinds), are connected in extremely regular ways, and usually simulate instantaneous transmissions between neurons.