# Why are activation functions independent layers in CNNs rather than part of convolutional layers?

I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization layers. A common normalization layer is a ReLU layer. But I never encountered an explanation of why all of a sudden, activation functions become their own layers in CNNs, while they are only parts of a fully connected layer in MLPs.

What is the reason for having dedicated activation layers in CNNs rather than applying the activation to the output volume of a convolutional layer as part of the convolutional layer, as it is the case for dense layers in MLPs?

I guess, in the end, there is no functional difference. We could just as well have separate activation layers in MLPs rather than activation functions in their fully connected layers. But this difference in the convention is irritating still. Well, assuming it only is an artifact of the convention.

• Hi and welcome to this community! Can you please cite an article that talks about normalisation layers in CNNs? – nbro Jul 20 '19 at 20:15
• Sure, this one for example: cs231n.github.io/convolutional-networks – lo tolmencre Jul 20 '19 at 20:18

$$\sigma \left(\mathbf{W} \mathbf{X} + \mathbf{b} \right)$$
where $$\sigma$$ is some activation function and $$\mathbf{W} \mathbf{X}$$ is the linear combination of the inputs, $$\mathbf{X}$$, and the weights, $$\mathbf{W}$$, and $$\mathbf{b}$$ is a bias. You could even represent a full or complete MLP (and not just one fully-connected layer) as a composite (or nested) function.