Let's say we are training a new neural network from scratch. I calculate the mean and standard deviation of my dataset (assume I am training a fully convolutional neural net and my dataset is images) and I standardise each channel of all images based on that mean and standard deviation. My output will be another image.

I want to use for example VGG for perceptual loss (VGG's weights will be frozen). Perceptual loss is when you input your prediction to a pretrained network to extract features from it. Then you do the same for the ground truth and the L2 distance between the features from ground truth and features from prediction is called perceptual loss.

As far as I know, I am supposed to standardise my data based on the mean and standard deviation VGG was trained with (since I am using VGG for inference essentially), which is different than the mean and standard deviation of my dataset. What is the correct way to do this? Should I undo the standardization of my dataset by multiplying standard deviation and adding the original mean to the output of my network, and then restandardise using VGG's statistics to calculate the loss? Or should I continue without restandardising?


I am a bit confused with the part of "training a new neural network from scratch" and then the part of "VGG's weights will be frozen" because the answer changes since they are different cases. It can also happen, from what I read, that you are using VGG as backbone of another network for transfer learning but it is not specified either. Anyway, I will try my best.

The cases I see:

  1. Training a network from scratch: no, you don't need to undo your normalization. In fact, when training from scratch you can normalize however you want.
  2. Training a network from scratch starting from pretrained weights: no, you don't need to change your normalization. Although it would help to faster convergence because the weights are fitted to other kind of normalization. But as the network trains it will fit to your new normalization without any problem.
  3. Training a network that uses VGG as backbone with frozen weights: no, you don't need to change your normalization although it would help. In spite of having your backbone fitted for a specific kind of normalization your network head is still an universal approximator that would fit to any other kind of normalization. However if you use the same normalization you are easing the network's job (that's why we normalize in the first place)

When you really need to force same normalization is in inference. When you want to infer you really need to do the same normalization in preprocessing or you won't have any meaningful results.

  • $\begingroup$ Sorry for not making it more clear. I am talking about using VGG as a loss function (aka perceptual loss). This is when you put the output of a network you are training through a pretrained VGG and get some features out of it from an intermediate layer. Then you do the same for your ground truth. Then your loss can be the L2 distance between the extracted features of your prediction and the extracted features of the ground truth $\endgroup$ – Andreas G. Dec 22 '20 at 13:56
  • $\begingroup$ I have modified the question to make it more clear $\endgroup$ – Andreas G. Dec 22 '20 at 14:03
  • $\begingroup$ Then yes, you are using VGG for inference (which in this case is a mean of computing the perceptual loss). You need to standardize your data according to however your VGG net was trained $\endgroup$ – JVGD Dec 22 '20 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.