I've never worked with very large models that require weeks or months of training, but in such a situation, what happens if you want to add extra features inputs, do you need to re-train the entire model again from scratch, or how is it handled? E.g. if Tesla added an extra sensor to its cars, would it need to re-train its network again from scratch to include this extra sensor input?
I'll try to explain how I would do it and the intuition behind it, feel free to correct me if something doesn't make sense. Lets consider you have an input of shape F where F is the number of features. If you were to construct a simple feed forward neural network you'll need to multiply the input with a weight matrix of shape (F, hidden_dim). Now if we want to add one more feature, the input will be of shape F+1 and the multiplication with the first layer will not work. What we could do to overcome this problem is to pad the weight matrix with an extra 0. In theory the new network should be able to reproduce the results of the first because it ignores the new feature. Now if we want to learn the "importance" of the new feature we could train the model to do so. I assume that the training time should be significantly lower since some of the weights might not need to be updated.