I am reading the paper Entropy-sgd: Biasing gradient descent into wide valleys by Chaudhari et al.

From what I understand, wide valleys tend to generalize better than sharp ones because they are likely representative of structures in the generating data. Whereas sharp ones could be anomalies that result from sampling when we create our training data set.

I don't understand how the following makes sense:

For an intuitive understanding of this phenomenon, imagine a Bayesian prior concentrated at the minimizer of the expected loss, the marginal likelihood of wide valleys under this prior is much higher than narrow, sharp valleys even if the latter are close to the global minimum in training loss.

So, if $p(\theta)$ is our prior distribution centered at the parameter which minimizes the training loss.

Then I think that it means that the integral of the probability over all the parameters that make up the wide valley would be greater than the integral of the probability over all the parameters that make up the narrow valley.

$$\int p(\theta_{\text{wide}})d\theta_{\text{wide}} > \int p(\theta_{\text{narrow}})d\theta_{\text{narrow}}$$

Which would make sense I suppose if all $\theta_{\text{wide}}$ have a reasonable training loss, which according to Deep Learning by Ian Goodfellow "for sufficiently large neural networks, most local minima have a low cost function value" pg. 278 it should. And and $\vert \theta_{\text{wide}} \vert >> \vert \theta_{\text{narrow}} \vert$.

I can see how this holds if we have a flat wide valley where all the points are minima, but what about a wide valley with a single minimum?

I would appreciate any explanation you can offer, as I am pretty confused, and there is a good chance I have gone off track.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.