How to explain that a same DNN model have radically different behaviours with each new initialization and training?

Question

I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. So the first $12$ elements of a data vector are all zeros except for one element which is equal to $1$ to denote the belonging of this data to a category. I'd like to add that my $X$ data is normalized with regard to the $13$-th column and that both $X$ and $y$ are shuffled in the same manner. Please find below my code for my model:

model = Sequential()
model.add(Input(shape=(13,)))
model.add(Dense(6, activation = 'relu'))
model.add(Dense(2, activation = 'relu'))
model.add(Dense(1))

model.compile(loss = 'mean_squared_error',
              optimizer = 'adam',
              metrics = ['RootMeanSquaredError'])

history = model.fit(X, y, validation_split = 0.1, epochs=64)

When trying to plot the learning curve using:

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('rmse')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

I get these curves:

There's already an "unusual" element to point here; I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner but we don't notice that on the training learning curve. For example the last four values of the loss are: $2.5176$, $3.4718$, $3.0704$ and it settles down on $3.8177$. I've also noticed that the losses provided by history.history are different than those shown during training, I suspect ones are computed before the epoch and ones after but I'm not sure.

I've tried to predict on the $275$ first elements of the training data. Most of the predictions took the value $4.2138872e+00$ but there are other predictions who took lesser values. I've computed the maximum of the predictions on the whole training set and it is $4.2138872e+00$.

I've also tried to train on the whole training set without a validation set to see what'll happen. I've made sure to rerun the cells of the model so that it doesn't take the weights it already found. I've noticed the same behaviour for the loss during training, but this time there is no constant predicted value that comes up as a maximum limit for the predictions.

I've already asked this question here and a user suggested to me that I should ask this question separately while providing the whole code. I ran the same code I was running and that was giving me the same predictions for no matter for my input vectors.

I think, as the user @Kostya that answered my previous question pointed out, what's happening here is called "dying ReLus". It's the same code that I'm running over and over but gives different predictions and the only random parameters are the weights and the biases. I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. I suppose they're randomly generated by a centered and reduced normal distribution.

I have last came to this question: does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ? I came to think that because if we had a large number of weights, their values are likely to fill that interval where the majority of the probability mass is concentrated. And since we have a small number of weights, we can get some "outlier" weights which lead to dyind ReLus.

Answer 1

I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.

Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (bias_initializer='zeros') and weights are initialized with Glorot uniform (kernel_initializer='glorot_uniform').

... "unusual" element to point here; I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner.

There's nothing unusual about the oscillations. Quite on the contrary - your curves are suspiciously too smooth.

I've also noticed that the losses provided by history.history are different

Yes, so this is a little gotcha in the keras implementation. For training loss, keras does a running average over the batches throughout an epoch, while for the validation loss it computes it after the epoch finishes. (link)

That also explains why your validation loss starts slightly lower than the training one - it actually lags behind the training loss by $\sim1/2$ of an epoch.

The fact that both losses stay almost equal suggests that your model don't really rely on training data for prediction - it got saturated ("dying ReLus") almost instantly.

Couple of suggestions that I can make:

• I don't see you setting up a learning rate. Try making it smaller (like, much smaller). And see if there's any difference.

• MSE is a nasty loss - especially at large values. Have you tried standardizing the target values?

Does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ?

Yes, the less neurons you have the higher the chance that all of the neurons will die out. You can get some intuition about it on https://playground.tensorflow.org/ - following the link, I've got a relu network that trains reasonably well. Try increasing the learning rate and observe saturation in the neurons. Reduce the number of neurons and see how the whole net gets stuck.