Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically argmin.

In ML, we simple use SGD with any initialization. But consider this: we take a few $n$ random combinations of weights and plot a visual graph (not to be confused with computation graph) where we find the local minimas of the graph (basically any point surrounded by larger point values would be minima). We store the weights (value of the variables in the polynomial) used for each point in the graph in a data structure.

Theoretically, if $n$ is big enough to be computationally efficient while being quite descriptive, we can simply take the weights of a random minima as initialization to the network and then perform SGD on it to converge to a global minima (hopefully).

This method would be quite faster since the initialization is better, and we don't need to compute for large values of $n$ - simply having a decent enough estimate. SGD would finally be used with a low learning rate to give the final push and we can be done with easier and faster.

So why don't we do this instead of having random initialization? is there theoretical basis on which this can't work?

  • $\begingroup$ AFAIK, Just Random Initialization (not your method) is not used, because it leads to either saturation or the variance in activations and gradients keeps decreasing. For reference $\endgroup$ May 30, 2021 at 5:53
  • $\begingroup$ Are you suggesting that you do some kind of up-front random parameter search of a limited number of parameters before training the model? It's an interesting idea, but I suspect the trade-off of how large $n$ needs to be would prevent this from being useful. $\endgroup$ May 31, 2021 at 9:52
  • $\begingroup$ @DavidHoelzer possibly yeah, but I expect the $n$ would have to be tuned as to be computationally viable - I see that it's much better to have a larger value of $n$ because substitution is much faster than backpropogating with SGD. Maybe, it might actually offset the time + computation because then SGD would converge extremely fast - but I guess it has to be tested. Maybe a good research topic, eh? I might try pitching it to professors to see what their opinions are about this. $\endgroup$
    – neel g
    May 31, 2021 at 11:26
  • $\begingroup$ I don't want to discourage you from it. It's interesting, but it feels like a monte carlo approach to possibly saving some time, but there's no guarantee that you're not still stuck in local minima. $\endgroup$ May 31, 2021 at 14:16


You must log in to answer this question.