0
$\begingroup$

I got this feedback for my thesis paper.

The improvement shown in the results section could be the result of random initialization. There should be multiple runs with means and standard deviations.

Can anyone explain this feedback with details?

I used a neural network with pre-trained weights for transfer learning (specifically, EfficientNetB0, with 'noisy-student' for the weights). It was a classification problem to classify between Covid-19, Viral Pneumonia, and normal cases. I normalised the dataset so that the images are in the range [0, 255] and I also did k-fold cross-validation.

$\endgroup$
0

1 Answer 1

3
$\begingroup$

Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split for cross-validation.

That means that any result metric from the neural network e.g. accuracy, loss, F1 score, is a random variable.

Reporting a single value of a random variable is not very informative. It may be higher or lower than expected, and it is difficult to tell if a result is significant.

Your reviewers are asking you to run the entire training and reporting from your dataset multiple times, to get the mean and standard deviation of any metrics that you reported.

You can keep the same train/test split for your hold-out test data set. Ideally this is the same train/test split as used by the classifiers that you are comparing your thesis results with.

If you used a seed for the RNG for repeatability, you can keep that as is and perform multiple training and reporting runs within a single script starting with the same seed set at the very beginning. Alternatively, if this would take too long, you could generate a set of seeds to use in consecutive runs - provided you are not selecting seed values depending on the results it does not matter.

$\endgroup$
1
  • $\begingroup$ Thank you so much Sir for your detailed answer. It helps me a lot. $\endgroup$ Commented Jul 5, 2021 at 7:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .