# Oscillating around the saddle point in gradient descent?

I was reading a blog post that talked about the problem of the saddle point in training.

In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to y at the saddle point, gradient descent will oscillate to and from the y direction. This gives an illusion of converging to a minima. Why is this?

Wouldn’t it continue down in the y direction and hence escape the saddle point?

Please go to Challenges with Gradient Descent #2: Saddle Points.

• IIRC, the main issue would be that in high dimensions there would be no "obvious" direction for the gradient descent to move towards, as you would tend to believe in a 3d graph like above. (some analogies fail in high dimensions) im not super certain about this answer (though i think its right), so i'll leave it as a comment instead Commented Mar 4, 2020 at 13:48
• Thank you for the reply. If there is no obvious direction then is it to say the saddle points flat area is relatively large in all directions? For a smaller flat area, if the y direction is the local minima in this saddle point then it would make sense to me for it to oscillate in that direction as it continues to overshoot the minima. It never considers the x direction as the gradients are too small. However, if the y direction is the local maxima here, then it should not be stuck in the y direction but x? Commented Mar 4, 2020 at 14:23
• yeah tbh i was also wondering about this question when i was first introduced to the problem. i think the point being is that in a pragmatic instance, your error surface won't be strictly-by-definition a saddle point with 0 gradient, but will be a small region with near-0 gradient. because there is small-noise in the gradient, the algo will wander around but cannot escape since there is no clear/strong tangental direction to escape the region with low gradient. actually i think this is the answer, so i'll write it properly Commented Mar 4, 2020 at 17:18
• fun fact: a friend told me once that in pragmatic instances (datasets of the "real world"), error surfaces might have an "egg carton shape" ("might", because what even does a high-dim egg carton look like?). we say this because we observe that (obviously: using the same ML architecture and hyper-parameters), we tend to get more or less the same error rates despite learning vastly different weights. the implication is that the error surface has many local minimas but they are more-or-less of "the same depth" Commented Mar 4, 2020 at 17:26
• I agree with k c . From my experience in running the same model many times the local minimum attained seem to result in about the same values for training accuracy with slightly more variance on validation accuracy. The saddle point issue is harder to get a handle on. As noted in other comments while mathematically a saddle point has a 0 value for the derivative in a real world model it might get very small but will not be zero. I would expect that the result would be oscillations in the flat area about the saddle point. Reducing the learning rate might help to converge to a smaller area Commented Mar 5, 2020 at 17:32