Weights initialization once the Neural Network is trained

Question

I am trying to understand how weights are initialized in a Neural Network using Keras deep learning framework and what happens if I train a Neural Network and then I want to train it again: are the weights of the previous training stored in some way and are the weights of the same Neural Network in the new training initialized based on the latest training ?

By "one training" I mean all the updates that the weights have received after all the epochs that are set.

To tell all the story: so far I have made my analysis assuming that with each new training (leaving the neural network architecture unchanged) the weights were initialized without taking into account the previous training. The concern come out when I wanted to use K-fold cross-validation procedure for which the dataset is split into K parts and K-1 parts are used for training while the remaining is left for testing. Iteratively this method uses all the K parts both for training and for testing. It is advised to do this in case of few data because in this way you can train the model K times each time with a different train/test split and you can produce significantly better estimates.

This can be achieved with the following code (I am dealing with natural language processing and so the code is based on those kinds of tasks):

#Whatever is needed to prepare data to be fed to our Neural Network

....

#Build keras model and fit

classes=2 # 3 or 4

embedding_dim = 10

X_train, X_test, y_train, y_test = train_test_split(padded_sequences, Y, test_size=.2)

# Define per-fold score containers 
acc_per_fold = []
loss_per_fold = []

# Merge inputs and targets
inputs = np.concatenate((X_train, X_test), axis=0)
targets = np.concatenate((y_train, y_test), axis=0)

num_folds=3

# Define the K-fold Cross Validator
kfold = KFold(n_splits=num_folds, shuffle=True)

# K-fold Cross Validation model evaluation
fold_no = 1
for train, test in kfold.split(inputs, targets):
    model = Sequential()#1.Define a model
    model.add(Embedding(len(tokenizer.word_index)+1, embedding_dim, input_length=max_length, name='embeddings')) 
    model.add(LSTM(64))
    model.add(Dense(1,activation='sigmoid')) 
    #Dense(classes,activation='softmax') for 4 classes
    print(model.summary())

    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
#sparse_categorical_crossentropy for 4 classes
    
    # Generate a print
    print('------------------------------------------------------------------------')
    print(f'Training for fold {fold_no} ...')
    
    model.fit(X_train, y_train,  batch_size=128,  validation_data=(X_test, y_test),verbose=2, epochs=15) 

    # Generate generalization metrics
    scores = model.evaluate(X_test,y_test,verbose=0)
    print(f'Score for fold {fold_no}: {model.metrics_names[0]} of {scores[0]}; {model.metrics_names[1]} of {scores[1]*100}%')
    acc_per_fold.append(scores[1] * 100)
    loss_per_fold.append(scores[0])
    
    # Increase fold number
    fold_no = fold_no + 1

After the first 15 epochs in which the Neural Network is trained on the first fold, the Neural Network is then trained on the second and again after 15 epochs it is trained on the third.

So, as said before, we do this because we can train the model K times each time with a different train/test split and you can produce significantly better estimates. So this assumes that after the first 15 epochs for the training on the first set of data, the weights are just updated towards better estimates in the second run of 15 epochs for the training on the second set of data and so on ?

In other words: if training K times is better than training once (in case of few data and under the limit of no-overfitting), does this mean that in some way the information of the previous training is stored and used for the second training and so on in order to be optimized more and more ?

In case of my specific example, I am interested in the weights of the embedding layer to have dense vector representation of my sequence of symbols. So I want to understand this for all the weights of a Neural Network in general but particularly on the embedding layers weights (I think that it is the same story for all the weights of all layers of the NN).

Anyway, besides the cross-validation procedure, if some kind of information is stored and kept from one training to the other, does it mean that each time I run my Neural Network training, this is influenced by the previous training ? For example, if I train this simple NN:

# define model
model = Sequential()
model.add(Dense(10, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
# fit the model
model.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)

First on a set of data and then on an other, in the second training with the new set of data, the weights are initialized on the basis of the previous training ?

On one side, this seems to me to have no sense and not to be possible, but on the other if it is not so, I do not find the meaning of using cross-validation technique (at least in the way it is done using Keras in the code I attached).

I tried to search what is the default initializer in Keras and I found this thread in Stack Overflow What is the default weight initializer in Keras?. I can find all the ways to initialize them but I am not able to find explanations about if these initializers take into account information from the previous trainings.

For example, as suggested in the answer to Stack Overflow question, for most of the layers, such as Dense, convolution and RNN layers, the default kernel initializer is 'glorot_uniform' and the documentation says that:

Draws samples from a uniform distribution within [-limit, limit], where limit = sqrt(6 / (fan_in + fan_out)) (fan_in is the number of input units in the weight tensor and fan_out is the number of output units).

But is this uniform distribution conditioned by the weights of the previous trainings ? Or by default this is not taken into account and there is some parameter to set to consider the previous ones ?

I think I miss something of important but I hope to have been clear and I thank you in advice.

Note: I took the code and emphasized sentence about cross-validation from this blog post: How to use K-fold Cross Validation with TensorFlow 2 and Keras?

Answer 1

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of each fold and validated on the validation data. Notice that this is exactly the idea behind cross validation - models trained on different folds are completely independent and they do not "communicate" in any manner. Should the models of the different sub-folds "communicated" and shared information between them, like, say, each model start training from the point the previous one finished, we would not talk about CV anymore (notice that such an approach could be valid if you purpose is something different than CV, but is is not a valid CV procedure).

The situation is slightly different in your 2nd code snippet; after it finishes, you could go on, either by running more epochs on the same data, or even training with different data. All you would need is to simply add model.fit statements sequentially:

# your 2nd snippet already run
# you can either continue with, say, 50 more epochs:
model.fit(X_train, y_train, epochs=50, batch_size=32, verbose=0)

# or continue training the model with different data, say X_train2 & y_train2:
model.fit(X_train2, y_train2, epochs=150, batch_size=32, verbose=0)

# of course you can do both, sequentially

What is the difference between your two code examples? In the first one, your model is redefined each time the loop executes; this defines a new model each time with the weights re-initialised, so no continuation of training. And, as already said, this is exactly the correct approach for cross validation.

But in the 2nd case, you have a single model, which you can just keep training sequentially (either with the same or even with different data, as shown above).

There are even cases where you would want to continue training the model outside of the current script; in such cases, you would save the model, and load it again later (or in another script altogether) in order to continue training it. This will work OK as long as you do not proceed to redefine/reinitalise the model. You can have a look at the documentation on Save and load Keras models for more.

I am not able to find explanations about if these initializers take into account information from the previous trainings.

They most certainly don't; if there is already existing information from previous runs (like in the cases I describe above), we do not even talk about weight initialization, i.e. there is no need for it - we already have the "initial" weights from the previous run. Random weight initialization in the sense discussed in the Stack Overflow thread you have linked above makes sense only for new models, like the ones produced in each run of the loop in your first code snippet above.