I was reading a blog post that talked about the problem of the saddle point in training.
In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to y at the saddle point, gradient descent will oscillate to and from the y direction. This gives an illusion of converging to a minima. Why is this?
Wouldn’t it continue down in the y direction and hence escape the saddle point?
Please go to Challenges with Gradient Descent #2: Saddle Points.