I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?

I found on https://towardsdatascience.com/natural-gradient-ce454b3dcdfa:

Instead of fixing the euclidean distance each parameter moves(distance in the parameter space), we can fix the distance in the distribution space of the target output.

Where did the distributions come from? If we wish to minimize f(x), the target output is just a minimizer x* right, and not a distribution, or am I missing something?

  • 1
    $\begingroup$ You're asking multiple distinct questions here. The question in the title is quite general and different from the question "Where did the distributions come from?". So, please, edit your post to leave only one question. $\endgroup$
    – nbro
    Mar 31 at 12:16
  • $\begingroup$ wiseodd.github.io/techblog/2018/03/14/natural-gradient/…. $\endgroup$ Mar 31 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.