I want to use deep learning to estimate the value of a function based on some data. However, the loss function would be neither convex nor concave. Can I know if it is a big deal in deep learning? Is training a deep network, when loss function is convex, the same as optimizing a convex problem or not? I would be thankful if any paper has addressed this issue.
In optimization, the loss function (sometimes called the error function) is a function that aggregates the disparity between actual and ideal behavioral states in multiple dimensions and over a sequence of input cases. In re-entrant (reinforced) learning, a feedback scalar or vector acts as a corrective signal that can replace or further aggregate with the loss function and additionally impact the training back-propagation.
In any of these architectures, including systems without NN components but attempt to adapt by converging on an optimal static or moving target behavior, one generality can be made. As the goal state is approached, the probability of successful convergence increases if the size of each incremental estimation decreases. The estimation is nothing more than an informed guess.
Decreasing the learning rate as the detected convergence value improves is one strategy being used experimentally if not in production, but that strategy has drawbacks when used alone. Involving the loss function in slowing down as the destination approaches is a best practice.
A biological system of a similar nature is the human subjective experience of pain. As the pain level goes down, the human brain cares less about the pain, therefore the steps taken to reduce it decrease and eventually vanish. Evolution has proven such to be advantageous for the same reason.
The Mathematics Involved
Maximizing the probability of an unknown function to be learned adequately to perform well is done in NNs by convergence. The term gradient descent is often used to describe the iterative process intended to converge on some ideal characterized by labeled data, a concept built into the network, or some fitness signal. The likelihood of converging on at least a local minima (which may or may not be the global minima) is much higher if the slope decreases as the minima is approached in successive approximation scenarios. This is when d2E / dt2 is positive.
The geometric idea of convexity is correlated to the calculus concept of the second derivative of a line or surface with respect to time or some other measure of forward progress. In successive approximation, the independent variable that measures forward progress could be time, computing cycles, the index of the training sample, iteration number, or some aggregation of these. The second derivative is the rate of the rate of change.
Convergence is more likely if the rate of change decreases as the disparity between what is perceived as optimal behavior and what is the actual current behavior. In other terms, the relationship between risk in making the next adjustment to circuit (NN) behavior should approach zero as proximity to the optimal behavior approaches zero. (If one senses they have worked there way to close proximity of their desired state, it makes no sense to make wild guesses.)
An Easy to Visualize Analogy
If you drop a rubber ball into a rigid cone, it will take time to reach thermodynamic rest, at the bottom. A lossless ball (considered impossible) will bounce forever. A paraboloid (parabolic in two dimensions like a solar reflector) will produce faster convergence with the same rubber ball because the ball drops in energy (the sum of kinetic and potential energy) with each bounce. The trajectory does not overshoot the bottom nearly as much or as frequently. This analogy is not perfect, but it provides a visual image without a diagram.
If you aggregate your disparity between your target trained behavior and the current in-training behavior in a way where the second derivative is negative (concave loss function with respect to distance) on either side of the targeted ideal, convergence is much less likely. In the analogy, the rubber ball is likely to bounce out of the flared cone altogether. A lossless ball will always bounce out eventually.
A more provincial analogy is that it would be like trying to catch a baseball with the back of a baseball glove.
Concave, Convex, or Zero Second Derivative
Whether continuous or discrete, convex functions converge much more frequently and usually with less time and computing resource than concave ones. A second derivative of zero is in the middle. The word linear is actually incorrect for this zero acceleration case. The correct term is first degree polynomial.
Sum of squares over the dimensions of the domain (inputs) and over the sequence of input cases will perform well in many cases. If you were to sum the square roots of the absolute value of error instead, your NN will rarely converge at all.
The following three things are more likely to be favorable if you find a way to aggregate your disparity between ideal and actual current behaviors in a way where the second derivative is greater than zero.
- Reliability of eventual convergence
- Speed of convergence
- Response time in the case of re-entrant (reinforced) learning
- Savings of computing cycles
- Reduction in the complexity of introspection
- Conservation of computational memory (RAM or SDD)
- Conservation of space needed for persistance and archiving
- Reduction of project cost to the business