1
$\begingroup$

Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks.

  1. First order gradient: It quantifies the rate of change of a function with respect to its inputs. It is useful in artificial intelligence, especially in gradient-based algorithms, to know about the direction in which the parameters need to be updated.

  2. Second-order gradient: It somehow quantifies the curvature of the function. It is used in artificial intelligence, to know whether the function has convex or concave portions.

In this context, I want to learn whether there is any significance for higher-order gradients in artificial intelligence? Note that higher-order refers to the order $\ge 3$.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Gradient descent presumes a Taylor Series. They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss.

The learner as a universal function approximator means there can be many configurations that yield minimum loss, and there is usually no global "best".

One of the reasons for multiple traverses through the data using the optimizer is that the local Taylor series estimate is low order (frequently only first order), so the optimum is not apparent after a single pass. When the weights change, the landscape changes and for a 25-million-parameter network, that change is happening in a 25-million dimensional space.

The higher order terms of the gradient can accelerate the optimization, but they can be prohibitively expensive to compute or estimate. The Hessian is nice, but has high overhead to compute and store. Things like conjugate gradient were popular because they are fast and (very) rough estimates of the Hessian.

If you have a cheap way to get decent higher-order derivatives, then there is value in that.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Conjugate gradient does not use an explicit Hessian, it is used for linear systems with symmetric system matrix, it only uses the evaluation procedure for the matrix-vector products, so can be used for large sparse systems where the matrix is never constructed as separate data structure. // Hessian approximations are constructed in quasi-Newton methods like BFGS. Often implemented in a way to avoid the full matrix, only storing the initial diagonal and the data for the rank-1-updates. $\endgroup$
    – Lutz Lehmann
    Commented Nov 29, 2022 at 9:06
  • $\begingroup$ @LutzLehmann - good points. I was meaning approximation in the loose and quick sense, so less bad than not having anything at all, and nothing like to within an error bound. If you have 25 million weights, then the hessian is a matrix of size (25M)^2. $\endgroup$
    – EngrStudent
    Commented Nov 29, 2022 at 16:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .