In the default implementation of batch normalization (e.g. the one in tensorflow), the training data is normalized according to the mean/variance of the current batch. Then in the so-called inference-mode, it uses the learned mean/variance (i.e. the moving mean/variance computed during training) to normalize the subsequent inputs.

I wonder if we can modify batch normalization in this way: during training, normalize according to the moving average/variance. This variant of 'batch normalization' would be equivalent to how the states/actions are frequently normalized in reinforcement learning.

Note that this behavior cannot be obtained using the tensorflow implementation, as the moving mean/variance are no longer updated in inference mode.

  • $\begingroup$ What's your question exactly? I understand that you're proposing some a variation of batch normalization, but I don't understand exactly what you're asking. Are you asking if it's theoretically sound? Or are you asking if it can be implemented in TensorFlow (and in this case your question would be off-topic)? Please, edit your post to clarify this and also write the title in the form of a question. $\endgroup$
    – nbro
    Mar 5 at 23:52


You must log in to answer this question.

Browse other questions tagged .