1
$\begingroup$

I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc.

A neural network is not a deterministic algorithm, so, in each training set, I train the neural network from scratch and I stop it when it's full converged.

After training is complete, I calculate the performance of the neural network in a test dataset. The problem is, I trained the neural network from scratch 2 times with the same parameters, but the difference in performance was almost 5%, which is a BIG DIFFERENCE.

So, what's the reasonable number of training runs to obtain a credible performance number of a neural network?

Improve this question
$\endgroup$
  • $\begingroup$ Are you shuffling the data before passing it to the net? Shuffling the data can cause these "issues" because that operation introduces randomness. $\endgroup$ – nbro♦ Jun 6 '20 at 12:44
  • $\begingroup$ No, I'm not shuffling. My neural network is a CNN and in each mini-batch i pad all images to have same size. So, all images in same mini batch has the size of the biggest image in that mini batch. But if i'm not shuffling, should not be the same in the 2 runs? $\endgroup$ – João Castilho Jun 6 '20 at 13:05
  • $\begingroup$ Are you using the same test set for both experiments? Also, have you tried to initialize the weights with the same values (not with the same strategy)? $\endgroup$ – nbro♦ Jun 6 '20 at 13:10
  • $\begingroup$ Yes, the trainning,validation and test sets are always the same. I'm initialize the weights with Glorot Uniform with default parameters $\endgroup$ – João Castilho Jun 6 '20 at 13:12
  • $\begingroup$ Initializing the weights with Glorot uniform may create a different set of initial weights. So, you may want to try initializing both neural networks with the exact same weights, to see if that solves the issue. $\endgroup$ – nbro♦ Jun 6 '20 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.