What would be the effect of normalizing each input patch going to Convolutional layer separately.

Let's say our input is 64 channels of the size 224x224 (like is the case for some hidden layers in ResNet). And instead of using Batch Normalization or Instance Normalization, we use "Patch Normalization". Our Convolutional layer is applying 3x3 kernel to the input. So we take each 64x3x3 input slice, calculate it's mean and std, use it to normalize, and then multiply it by the layer weights.

It's rather obvious idea, so I assume it was tested multiple times. But I was unfortunate to find any info about it.

  • $\begingroup$ Interesting...implement and see the results... you can utilize some tool to visualize what kernels are learning and compare that with kernels in case of BatchNormalization. Implement and post results side by side. $\endgroup$ – MANU Jun 19 '20 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.