Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically argmin.
In ML, we simple use SGD with any initialization. But consider this: we take a few $n$ random combinations of weights and plot a visual graph (not to be confused with computation graph) where we find the local minimas of the graph (basically any point surrounded by larger point values would be minima). We store the weights (value of the variables in the polynomial) used for each point in the graph in a data structure.
Theoretically, if $n$ is big enough to be computationally efficient while being quite descriptive, we can simply take the weights of a random minima as initialization to the network and then perform SGD on it to converge to a global minima (hopefully).
This method would be quite faster since the initialization is better, and we don't need to compute for large values of $n$ - simply having a decent enough estimate. SGD would finally be used with a low learning rate to give the final push and we can be done with easier and faster.
So why don't we do this instead of having random initialization? is there theoretical basis on which this can't work?
