I'm trying to perform a segmentation task on images of multiple sizes using fully convolutional neural networks.
Currently, I'm using EfficientNet as a feature extractor, and adding a deconvolution/backwards convolution/transposed convolution layer as described in the original Fully Convolutional Networks for Semantic Segmentation paper.
But this transposed convolution layer doesn't return a filter of a size equivalent to the original image for images of varying sizes.
For example, let's assume the original image is $100 \times 100$, and the last layer contains filters of size $50 \times 50$. To get a filter of the same size as the original, you would need a transposed convolution layer of size $51 \times 51$.
Now, assume you passed in an image of size $200 \times 200$. The last layer would contain filters of size $100 \times 100$. That same transposed convolutional filter of size $51 \times 51$ would result in an output of size $150 \times 150$.
Is there any way to make it so that a fully convolutional network always returns an image of the same size as the original?