Why does a fully connected layer only accept a fixed input size?

Question

I'm studying how SPP (Spatial, Pyramid, Pooling) works. SPP was invented to tackle the fix input image size in CNN. According to the original paper https://arxiv.org/pdf/1406.4729.pdf, the authors say:

convolutional layers do not require a fixed image size and can generate feature maps of any sizes. On the other hand, the fully-connected layers need to have fixed size/length input by their definition. Hence, the fixed size constraint comes only from the fully-connected layers, which exist at a deeper stage of the network.

Why does a fully connected layer only accepts a fixed input size (but convolutional layers don't)? What's the real reason behind this definition?

Answer 1

A convolutional layer is a layer where you slide a kernel or filter (which you can think of as a small square matrix of weights, which need to be learned during the learning phase) over the input. In practice, when you need to slide this kernel, you will often need to specify the "padding" (around the input) and "stride" (with which you convolve the kernel on the input), in order to obtain the desired output (size). So, even if you receive inputs of different sizes, you can change these values, like the padding or the stride, in order to produce a valid output (size). In this sense, I think, we can say that convolutional layers accept inputs of (almost) any size.

The number of feature maps does not depend on the kernel or input (sizes). The number of features maps is determined by the number of different kernels that you will use to slide over the input. If you have $K$ different kernels, then you will have $K$ different feature maps. The number of kernels is often a hyper-parameter, so you can change it (as you please).

A fully connected (FC) layer requires a fixed input size by design. The programmer decides the number of input units (or neurons) that the FC layer will have. This hyper-parameter often does not change during the learning phase. So, yes, FC often accept inputs of fixed size (also because they do not adopt techniques like "padding").

Answer 2

It doesn't have to be so. Fully connected layer could be considered as convolutional layer with input image of 1 pixel and spatial kernel size of 1 pixel. So 1-pixel kernel convolutional layer is effectively the same as fully connected layer attached to each pixel. That is idea behind "Fully Convolutional Networks". If you want "true", 1-pixel fully connected layer after convolutional layer (with variable input size) all you have to do is to put average pooling layer (or other type of pooling layer) before fully connected layer. That way fully connected layer can accept variable input size.