Quick question about LLMS (and gradient descent in general): we search the space of neural networks by gradient descending in order to minimize one explicit function but what seems to be happening is that in the course of minimizing this function, the neural network automatically picks up several other skills (like building a world model...?). I imagine there is a lot of randomness in this process so to what extent are the extra skills picked up "fixed".
In other words, if I train models multiple times on the same data using the same loss function, to what extent are the resulting neural networks similar in performance (out of the training set, say)? Does the answer to this question matter much on what the loss function is and how much training has taken place?
Given the proliferation of LLMs with not too dissimilar behaviour, I expect the answer to the above question is positive (maybe in the limit that the training time tends to infinity). But theoretically, this is a little surprising to me that there is an "almost unique" minimizer of the given loss function. Do we have a good theoretical framework for explaining this?