1
$\begingroup$

I am interested in your opinion on the topic if you think that it makes sense to use batch normalization layer in a network that is trained with a batch size of 1. This is a special case as part of an experiment. What effects can be expected?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

It's not possible to use batch normalization with a batch size of 1. Batch normalization requires you to calculate the variance of activation values in the current batch, and variance is undefined for a sample of size 1.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Why it is undefined? It is zero. $\endgroup$
    – Antonios Sarikas
    Commented Feb 15 at 17:54
  • 1
    $\begingroup$ The sample variance formula has a denominator of N - 1. When N = 1, N - 1 = 0, and division by 0 is undefined. @adosar $\endgroup$
    – Johann
    Commented Feb 18 at 0:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .