Do gradient-based algorithms deal with the flat regions with desired points?

Question

I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points.

For example, let us consider a function whose local/global minimum or maximum values lies on flat regions. You can take this graph (just) for example.

Can gradient-based algorithms work on those curves with their local/global minima, or do maxima lie on flat regions?

Answer 1

Can gradient-based algorithms work on those curves with their local/global minima, or do maxima lie on flat regions?

Yes, with some minor caveats.

All the points on the flat region are equivalent (and in your example, are all valid global minimum points). Gradients outside of the region will point correctly away from that region and gradient descent steps will therefore move parameters towards it.

Provided the step size multiplied by the gradient near the flat region is not too large, then a step taken near it will end up with parameters inside the region. After that, then any further gradients will be zero, so it is not possible to use basic gradient steps to escape it.

In the case of a global minimum, that's fine, you don't care which point in the global minimum you have converged to (otherwise your function to optimise would be different).

In the case of local minima or saddle points, you might care to use optimisation methods that can escape flat areas. Minibatch or stochastic gradient descent can do this because gradient measurements are noisy, whilst momentum algorithms can continue making steps when the immediate gradient signal is zero.

The example function you used is not something you would expect to come across when optimising a machine learning algorithm, although some loss functions do have components that have similar behaviour. For example, triplet loss uses a $\text{max}(d_1 - d_2 + \alpha, 0)$ where $d_1$ and $d_2$ are distances between an anchor image and desired class versus different class respectively, and $\alpha$ is a margin or minimum desirable distance between classes. The details of this are not important unless you want to create a face recogniser or similar - the important detail for your question is that $\text{max}(x, 0)$ is really used in ML as a loss function, and may have a similar shape to your example function. Once used in aggregate with many data examples though, and with regularisation, the shape would not be so simple, and proabably would not have any reachable flat minima regions like this.