0
$\begingroup$

It is said that a fully convolutional network can handle any image size. I don't understand.

Unlike regular CNN, a fully convolutional network reinterprets the dense layer as a convolutional operation. Specifically, we think of it as a list of kernels, each having the same size as the input (a.k.a. the activations of the previous layer) during training. This is fine if the input image at test time is larger, because the input size will be bigger, and the kernel can slide around. But it seems a problem if the image is smaller, as the input size will be smaller than the kernel.

So I wonder how a fully convolutional network can handle smaller images.

Improve this question
$\endgroup$
2
  • $\begingroup$ Do these posts 1 and 2 answer your question? $\endgroup$
    – Luca Anzalone
    Commented May 1, 2023 at 12:57
  • $\begingroup$ No, I had read both before I wrote this question. I also read the original paper. $\endgroup$
    – Tom Bennett
    Commented May 1, 2023 at 13:29

1 Answer 1

Reset to default
0
$\begingroup$

Fully convolutional networks (FCNs) replace dense layers with $1\times 1$ convolutional layers in order to handle different input sizes: as $K\times K$ kernels are applied to spatial locations that can vary in number, instead a dense weight matrix has a fixed shape that can't adapt to a different input dimensionality.

Now, what about an input image that is smaller than the ones used for training?

I think the elements to consider are the following: kernel size, padding, strides and pooling layers. Considering a fixed kernel size $K$ for each convolutional layer, each time we apply a conv the activation volume will be smaller in size (width and height) unless padding is used. The same is true for strided convolutions and pooling layers: they shrink the output by a given factor (e.g. $2$).

Suppose there are $L$ conv layers (without padding), and $P$ pooling layers with stride $2$, the total stride of the model is: $S=(K\times L) + 2^P$. This means that, in principle, $H\times W$ input images should be larger than the total stride, i.e. $H\ge S+1$ and $W\ge S+1$ (indeed, considering appropriate rounding to the integer part.)

So the penultimate activation map should be at least $1\times 1$ in size in order to be processed by the last point-wise convolution that makes dense predictions.

NOTE: I haven't verified this in practice, but I want to provide some intuitions and these are indeed practical constraints in order for convolutions or pooling to work at all: you cannot slide a kernel on an image that is smaller than the kernel itself. So, extending this to multiple layers you obtain $S$ (that I call the total stride, but I don't know if there is a proper name for this.)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the response. I don't think FCN replaces dense layers with 1x1 convolutions where the kernel size is $1\times1\times D$. Instead, it replaces dense layers with kernels that have the same dimension as its input (dim of the previous layer) during training, such that the output of such layer is $1\times1\times N$. This means the previous layer would have to be as large as the kernel. Maybe the input image that are too small will have to be padded or resized during testing? $\endgroup$
    – Tom Bennett
    Commented May 1, 2023 at 17:37
  • $\begingroup$ Sure, padding or resizing will do the job. Probably for very small images you can up-sample them, forward the model, then down-sample the segmentation masks and apply them to the original image. $\endgroup$
    – Luca Anzalone
    Commented May 1, 2023 at 20:16
  • $\begingroup$ It feels odd that I haven't seen this problem mentioned elsewhere. $\endgroup$
    – Tom Bennett
    Commented May 2, 2023 at 0:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .