3
$\begingroup$

Andrej Karpathy said he's shot himself in foot over and over gain with batch norm.

I mentioned no one likes this layer. It causes a huge amount of bugs and intuitively it's because it is coupling examples in the forward pass of a neural net and I've shot myself in the foot with this layer over and over again in my life and I don't want you to suffer the same so basically try to avoid it as much as possible. Some of the other alternatives to these layers are for example group normalization or layer normalization.

How exactly can you shoot yourself in the foot with batch norm?

$\endgroup$

1 Answer 1

3
$\begingroup$

I think the problems with batch normalization arise mainly due to the large misunderstanding of what it does, combined with a simple but confusing name.

Despite the simple computation (everybody knows what normalization is), it is not so clear why batch normalization provides the benefits it does. In the original paper the main explanation given was that batch normalization reduced covariat shift for the parameters in very deep layers, but there are many papers out there who proved already that batch normalization also smooth classic loss functions, which means it can be considered as a regularization layer. And indeed some papers pointed out that combining batch norm with other regularization techniques like dropout doesn't really provide any benefit. You can find all the papers link in the first answer to this question.

So this already answer how you can shoot yourself in the foot with batch normalization, said that it's hard to say a priori when you should use it or not, unfortunately like many other aspects of deep learning this is still a matter of trials and errors.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .