3
$\begingroup$

I want to see the effects of changing some training parameters (batch size, learning rate, optimizer...) to the accuracy obtained. The problem is that with the same parameters I get significantlly different results (up to 5%).

I load the same weights before training and I deactivated the shuffle. To my understanding this should be enough to get the same results. The only randomness of the backpropagation comes from the initial weight values, right? What am I missing?

This is how I compiled the model:

model.compile(loss="categorical_crossentropy",
              optimizer=keras.optimizers.SGD(lr=0.1, decay=1e-6),
              metrics=['accuracy'])

And this is the code that I repeat:

model.load_weights("./saved_weights.h5", by_name=False)

model.fit(x_train, y_train,
          batch_size=32,
          epochs=1,
          initial_epoch=0,
          validation_data=(x_test, y_test),
          shuffle=False)

I'm using tensorflow 2.0 as a backend in a colabs's CPU.

Improve this question
$\endgroup$
  • $\begingroup$ Questions related to programming issues are off-topic here. Please, read our on-topic page ai.stackexchange.com/help/on-topic to understand which questions you can ask here. If you have a programming issue, the best place to ask your question is probably Stack Overflow or Data Science SE. $\endgroup$ – nbro♦ Jun 17 '20 at 14:31
4
$\begingroup$

As @codeblooded said, you should set random seed for numpy and keras, and also set pythonhashseed. The seeds set the state of the random number generator which makes the results different.

This method only works when you train the network on CPU. The problem with getting same result on GPU every single time is that cuDNN is not deterministic. Specifically, the reduce_sum() function in tensorflow is known to have different results even if a random seed is set. You can see this for more details. But simply said, reduce sum is not deterministic in tensorflow, but however it is used in bias gradient calculation. This can be avoided by using matmul in place of reduce sum, and you can see the implementation in the article mentioned.

To get reproducible results, you can try using different framework, like pytorch. I have done experiments on pytorch and it seems to be reproducible. See this colab notebook. All you need to do is add a couple lines to the code. Theano based keras seems to work as well but I haven't tested it.

However for hyperparameter testing and searching 0.3% should not affect the result, and if you really want very accurate result, average the accuracy of 30 or more tries to get an accurate result. If you do this you don't need to set the random seed.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thanks! I'll definitely test the hyperparameters with de ~0.3%. But I also want to understand how NN training works and this isn't working as I expected. I'm using colab without hardware acceleration i.e. GPU or TPU, so cuDNN shouldn't be the problem. $\endgroup$ – David Rubio Jan 8 '20 at 13:56
  • 1
    $\begingroup$ I am not sure... maybe some functions isn't deterministic on cpu as well in tensorflow. If you want exact results you can try using other frameworks or use theano backed for keras. $\endgroup$ – Clement Hui Jan 8 '20 at 13:58
  • 1
    $\begingroup$ I'll try theano and I'll report back the results. $\endgroup$ – David Rubio Jan 8 '20 at 14:01
  • $\begingroup$ It looks like the seeds in Theano are way more complex than in TF, NumPy or Python. Without setting up the seed theano also gives different results each time. $\endgroup$ – David Rubio Jan 8 '20 at 15:14
  • $\begingroup$ Oh.. well then maybe you can try using pytorch. It is easy to use and many researcher uses it. Pytorch is more "pythonic" than tensorflow or keras and is a great framework. $\endgroup$ – Clement Hui Jan 8 '20 at 15:16
3
$\begingroup$

See comprehensive answer here; to paste a snippet, below is complete code for fixing a random seed:

def reset_seeds(reset_graph_with_backend=None):
    if reset_graph_with_backend is not None:
        K = reset_graph_with_backend
        K.clear_session()
        tf.compat.v1.reset_default_graph()
        print("KERAS AND TENSORFLOW GRAPHS RESET")  # optional

    np.random.seed(1)
    random.seed(2)
    tf.compat.v1.set_random_seed(3)
    print("RANDOM SEEDS RESET")  # optional

The TF graph must be reset, else resetting the TF random seed won't take full effect. K is the backend, for example import keras.backend as K or import tensorflow.keras.backend as K

Further reading:

UPDATE: your case is actually unique relative to my linked answers; load_weights is not sufficient to restore your model's state - you must also load the optimizer.

All Keras optimizers have self.weights, self.updates, and self.iterations attributes, which are stateful - i.e. depend on the past history of iterations. For SGD without momentum, the former two shouldn't matter (am uncertain on this), but with decay!=0, self.iterations absolutely does, for this line of code. If you don't load the optimizer state, then self.iterations is reset to zero, thus your actual learning rate may differ significantly from the originally saved model's.

To remedy, you can either save everything via model.save(include_optimizer=True), or save the optimizer and its self.iterations attribute separately. Luckily, I do this all the time, and use my own custom method; in its current shape, it's not the most user-friendly, but I can polish it up and share it if interested.

Improve this answer
$\endgroup$
  • $\begingroup$ It didn't work better than the other answers, but thanks for the comprehensive answer. I'm still wondering why loading the same weights before training isn't enough. Shouldn't this make the seed-setting irrelevant? Aside from initializing the weights, where is the randomness used during training? $\endgroup$ – David Rubio Jan 9 '20 at 9:05
  • $\begingroup$ I think I found why it happens. The atomicAdd() function in NVIDIA CUDA is the origin of the problem. @David Rubio please read the discussion in chat. $\endgroup$ – Clement Hui Jan 9 '20 at 14:04
  • $\begingroup$ @DavidRubio Without seeing your full model and data pipeline code, can't tell much further; one existing unresolved problem is with stacking CNNs - so if you have conv layers, you share my problem. -- The answer to your comment's question is in the second link above, under "Sources of randomness" $\endgroup$ – OverLordGoldDragon Jan 9 '20 at 14:55
  • $\begingroup$ @DavidRubio Also, loss is a superior measure of non-reproducibility, as accuracy is lot more susceptible to small differences; a .1% confidence difference is enough to drive accuracy from 100% to 0% on a sample, but loss will barely budge. $\endgroup$ – OverLordGoldDragon Jan 9 '20 at 14:57
  • $\begingroup$ @DavidRubio Now that I've actually fully read your question, my original answer missed something obvious - see updated. $\endgroup$ – OverLordGoldDragon Jan 9 '20 at 18:24
2
$\begingroup$

Use seed for random functions.

For example if you are using numpy random function

from numpy.random import seed
seed(1)

Read more about reproducible results here, https://machinelearningmastery.com/reproducible-results-neural-networks-keras/

Set PYTHONHASHSEED environment variable at a fixed value

import os
os.environ['PYTHONHASHSEED'] = str(1)

https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Just to elaborate a bit, the weights and biases are random and can result in significant differences in training results aside from the hyper parameter tuning that you are working on. $\endgroup$ – David Hoelzer Jan 8 '20 at 11:58
  • $\begingroup$ I've used numpy.random.seed(1) and tf.compat.v2.random.set_seed(2) as suggested in the link and the results are getting much closer (arround 0.3%), but still not the same. This is odd. $\endgroup$ – David Rubio Jan 8 '20 at 12:16
  • $\begingroup$ @DavidHoelzer Exactly but I'm trying to use the same initial weights and biases by loading the same ones each time. It seems that it doesn't work, but I don't know why. $\endgroup$ – David Rubio Jan 8 '20 at 12:22
  • 1
    $\begingroup$ dropout also contains randomness $\endgroup$ – Clement Hui Jan 8 '20 at 12:26
  • 1
    $\begingroup$ CUDNN also have some randomness. see this: github.com/keras-team/keras/issues/2479#issuecomment-213987747 $\endgroup$ – Clement Hui Jan 8 '20 at 12:27
0
$\begingroup$

In Tensorflow 2.0 you can set random seed like this

import tensorflow as tf
tf.random.set_seed(221)


from tensorflow import keras
from tensorflow.keras import layers


model = keras.Sequential( [ 
layers.Dense(2,name = 'one'),
layers.Dense(3,activation = 'sigmoid', name = 'two'),
layers.Dense(2,name = 'three')])

x = tf.random.uniform((12,12))
model(x)
Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.