Why are activation functions independent layers in CNNs rather than part of convolutional layers?

Question

I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization layers. A common normalization layer is a ReLU layer. But I never encountered an explanation of why all of a sudden, activation functions become their own layers in CNNs, while they are only parts of a fully connected layer in MLPs.

What is the reason for having dedicated activation layers in CNNs rather than applying the activation to the output volume of a convolutional layer as part of the convolutional layer, as it is the case for dense layers in MLPs?

I guess, in the end, there is no functional difference. We could just as well have separate activation layers in MLPs rather than activation functions in their fully connected layers. But this difference in the convention is irritating still. Well, assuming it only is an artifact of the convention.

Answer 1

These are just two equivalent interpretations (or illustrations) of the application of an activation function. In other words, in a multi-layer perceptron (MLP), you could also illustrate the application of the activation function as a separate layer that follows a linear combination layer. However, in the context of MLPs, the math is relatively simple and elegant, so a fully-connected layer of an MLP can simply be represented as follows

$$ \sigma \left(\mathbf{W} \mathbf{X} + \mathbf{b} \right) $$

where $\sigma$ is some activation function and $\mathbf{W} \mathbf{X}$ is the linear combination of the inputs, $\mathbf{X}$, and the weights, $\mathbf{W}$, and $\mathbf{b}$ is a bias. You could even represent a full or complete MLP (and not just one fully-connected layer) as a composite (or nested) function.

In the context of convolutional neural networks (CNNs), people might illustrate the application of the activation function as a separate layer because the application of an activation function to the result of the convolution operation of the CNN is optional and out-of-favor (as stated in this article http://cs231n.github.io/convolutional-networks), as opposed to the case of MLPs, where activations functions usually follow the linear combination. However, note that the last layers of a CNN are usually fully-connected layers (and not convolutional or pooling layers), that is, they are a linear combination of their input and their weights followed by an application of an activation function.