I'm facing the problem of having images of different dimensions as inputs in a segmentation task. Note that the images do not even have the same aspect ratio.

One common approach that I found in general in deep learning is to crop the images, as it is also suggested here. However, in my case I cannot crop the image and keep its center or something similar since in segmentation I want the output to be of the same dimensions as the input.

This paper suggests that in a segmentation task one can feed the same image multiple times to the network but with a different scale and then aggregate the results. If I understand this approach correctly, it would only work if all the input images have the same aspect ratio. Please correct me if I am wrong.

Another alternative would be to just resize each image to fixed dimensions. I think this was also proposed by the answer to this question. However, it is not specified in what way images are resized.

I considered taking the maximum width and height in the dataset and resizing all the images to that fixed size in an attempt to avoid information loss. However, I believe that our network might have difficulties with distorted images as the edges in an image might not be clear. What is possibly the best way to resize your images before feeding them to the network?

Is there any other option that I am not aware of for solving the problem of having images of different dimensions?

Also, which of these approaches you think is the best taking into account the computational complexity but also the possible loss of performance by the network?

I would appreciate if the answers to my questions include some link to a source if there is one. Thank you.


I will give a more thorough answer.

There are 2 problems you might face.

1) Your neural net (in this case convolutional neural net) cannot physically accept images of different resolutions. This is usually the case if one has Fully-Connected layers, however if the network is Fully-Convolutional then it should be able to accept images of any dimension. Fully-convolutional implies that it doesn't contain fully-connected layers, but only convolutional, max-pooling and batch normalization layers all of which are invariant to the size of the image. Exactly this approach was proposed in this ground-breaking paper Fully Convolutional Networks for Semantic Segmentation. Keep in mind that their architecture and training methods might be slightly outdated by now. Similar approach was used in widely used U-Net: Convolutional Networks for Biomedical Image Segmentation and many other architectures for object detection, pose estimation and segmentation.

2) Convolutional neural nets are not scale invariant. For example if one trains on the cats of the same size in pixels on images of fixed resolution, the net would fail on images of smaller or larger sizes of cats. In order to overcome this problem, I know of two methods (might be more in the literature): 1) multi-scale training of images of different sizes in fully-convolutional nets in order to make the model more robust to changes in scale; and 2) having multi-scale architecture. A place to start is to look at these two notable papers: Feature Pyramid Networks for Object Detection and High-Resolution Representations for Labeling Pixels and Regions.


Assuming you have a large dataset, and it's labeled pixel-wise, one hacky way to solve the issue is to preprocess the images to have same dimensions by inserting horizontal and vertical margins according to your desired dimensions, as for labels you add dummy extra output for the margin pixels so when calculating the loss you could mask the margins.

  • $\begingroup$ How does one deal with normalization then in these cases? Do you only normalize the pixels in an image that are not included in its margin I guess? $\endgroup$ – MattSt May 5 '18 at 10:30
  • $\begingroup$ Yes, because your data generating process is having different sizes so if you include the margins, you will change the data distribution. margins are inserted to group the training samples into batches because your problem needs to output a fixed output vector. $\endgroup$ – Fadi Bakoura May 5 '18 at 14:57

I think this paper will have useful insights for you.


As you want to perform segmentation, you can use U-Net. It does not have Fully Connected Units. Therefore, size of input will not matter.

  • 2
    $\begingroup$ I think you should elaborate on your points a little bit more. $\endgroup$ – DuttaA Jan 28 '19 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.