I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?
Instead of fixing the euclidean distance each parameter moves(distance in the parameter space), we can fix the distance in the distribution space of the target output.
Where did the distributions come from? If we wish to minimize f(x), the target output is just a minimizer x* right, and not a distribution, or am I missing something?