For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.
The loss function is sometimes called an error function. Its inverse is a wellness or value function.
The intention of each iteration is to decrease the result of applying the loss function. The particular method intended to produce this decrease is to calculate the gradient of the loss function and use it to compute the incremental change in parameters likely to reduce the loss. It is often used in conjunction with back propagation to distribute the corrective signal over a sequence of layers, each of which is parameterized.
To avoid overshooting the optimum, leading to oscillation or chaotic behavior, the corrective signal is attenuated with a factor called the learning rate. Too low a learning rate will compromise the speed of convergence.
Several strategies exist to calculate loss functions, hyper-parameterize the corrective signaling that is back-propagated, or integrate other search strategies to improve either reliability, speed, or accuracy.
