This question considers the convergence of an artificial networks (MLPs, RNNs, LSTM nets, CNNs) over time or over the course of epochs made up of iterations through training examples. In this question's context, we can simplify the correspondence of time and iteration number. We can assume that Δt is proportional to iteration number and proportional to epoch number, so that, for simplicity's sake, iteration awareness means the same thing as temporal awareness.
Isaac Newton published a method for estimation using N terms of finite differences in his 1687 Principia Mathematica. It is essentially the discrete version of the Taylor Series Expansion that that 12th grade students or first year college students learn.
When the Jacobian (second year calculus) is used to perform gradient descent so that an artificial network can converge on the objective being learned, that is the application of the first two terms of Newton's formula applied to the number of freedoms of motion in the back propagation.
The formulas for each iteration in training can be easily derived from the basics of interpolation and knowledge of college calculus. There is nothing to the math that could not be easily grasped by students in the early 18th century that studied mathematics or science at the university level.
That each activation layer must be factored in is a new combination of century old concepts, and that particular combination of concepts (along with von Neumanm computers fast enough to try it) was the key to developing a practical multilayer perceptron that converges.
I am puzzled by the lack of the use of some obvious facts about knowledge acquisition in humans when creating these artificial structures.
- When we decide to adjust our view of the world, we don't just consider what we think now, but also what we used to think.
- When a view is not brought into consciousness, we loose the strength of the view, which is probably an evolutionary advantage since the view may become obsolete over time.
- When we see that we repeatedly discover new reasons to change our view in a particular direction, we tend to increase the size of our adjustments to our view.
In the simplistic modelling of learning inherent in an artificial network, we have one of the three aspects of learning appearing in extensions of the multilayer perceptron concept. We see the concept of using past state in recurrent artificial networks (RNNs).
We don't see natural (inverse exponential) decay functions that tend old parameters toward their neutral values. Or have I missed some research?
The interpolation beyond the Jacobian (two terms) and Hessian (three terms) is not used for reasons of computational burden, which I understand, but why not use the previous states in normal Newtonian fashion to go out one or two more terms. Or have I missed that research?
Has anyone tried using two dimensional lookup tables with interpolation to avoid the computational burden of additional terms in descent? Again, this only requires 18th century Newtonian interpolation. Did I miss that research too?
My understanding is that there has been work on hyper-parameters to dampen back propagation feedback. Dampening feedback only makes sense if the adjustments appear chaotic. However, shouldn't adjustment of parameters be augmented rather than dampened if the convergence adjustment appears to be consistent or increasing with each iteration?
I know of LSTM and the newer attention based research, but neither of these really address the above questions and potentially advantageous convergence ideas to my knowledge. Again, there may be work over which I have not yet stumbled. Who, if anyone, is thinking along any of these somewhat intuitive lines?
Please provide references to books, papers, reports, or articles so that we can all be edified.