# Convolutional neural networks with input images of different dimensions - Image segmentation

I'm facing the problem of having images of different dimensions as inputs in a segmentation task. Note that the images do not even have the same aspect ratio.

One common approach that I found in general in deep learning is to crop the images, as it is also suggested here. However, in my case I cannot crop the image and keep its center or something similar since in segmentation I want the output to be of the same dimensions as the input.

This paper suggests that in a segmentation task one can feed the same image multiple times to the network but with a different scale and then aggregate the results. If I understand this approach correctly, it would only work if all the input images have the same aspect ratio. Please correct me if I am wrong.

Another alternative would be to just resize each image to fixed dimensions. I think this was also proposed by the answer to this question. However, it is not specified in what way images are resized.

I considered taking the maximum width and height in the dataset and resizing all the images to that fixed size in an attempt to avoid information loss. However, I believe that our network might have difficulties with distorted images as the edges in an image might not be clear. What is possibly the best way to resize your images before feeding them to the network?

Is there any other option that I am not aware of for solving the problem of having images of different dimensions?

Also, which of these approaches you think is the best taking into account the computational complexity but also the possible loss of performance by the network?

I would appreciate if the answers to my questions include some link to a source if there is one. Thank you.

## 6 Answers

I will give a more thorough answer.

There are 2 problems you might face.

1) Your neural net (in this case convolutional neural net) cannot physically accept images of different resolutions. This is usually the case if one has Fully-Connected layers, however if the network is Fully-Convolutional then it should be able to accept images of any dimension. Fully-convolutional implies that it doesn't contain fully-connected layers, but only convolutional, max-pooling and batch normalization layers all of which are invariant to the size of the image. Exactly this approach was proposed in this ground-breaking paper Fully Convolutional Networks for Semantic Segmentation. Keep in mind that their architecture and training methods might be slightly outdated by now. Similar approach was used in widely used U-Net: Convolutional Networks for Biomedical Image Segmentation and many other architectures for object detection, pose estimation and segmentation.

2) Convolutional neural nets are not scale invariant. For example if one trains on the cats of the same size in pixels on images of fixed resolution, the net would fail on images of smaller or larger sizes of cats. In order to overcome this problem, I know of two methods (might be more in the literature): 1) multi-scale training of images of different sizes in fully-convolutional nets in order to make the model more robust to changes in scale; and 2) having multi-scale architecture. A place to start is to look at these two notable papers: Feature Pyramid Networks for Object Detection and High-Resolution Representations for Labeling Pixels and Regions.

Assuming you have a large dataset, and it's labeled pixel-wise, one hacky way to solve the issue is to preprocess the images to have same dimensions by inserting horizontal and vertical margins according to your desired dimensions, as for labels you add dummy extra output for the margin pixels so when calculating the loss you could mask the margins.

• How does one deal with normalization then in these cases? Do you only normalize the pixels in an image that are not included in its margin I guess? – MattSt May 5 '18 at 10:30
• Yes, because your data generating process is having different sizes so if you include the margins, you will change the data distribution. margins are inserted to group the training samples into batches because your problem needs to output a fixed output vector. – Fadi Bakoura May 5 '18 at 14:57

I think this paper will have useful insights for you.

• You are not supposed to give link-only answers. – JohnnyApplesauce Apr 11 at 14:53

As you want to perform segmentation, you can use U-Net. It does not have Fully Connected Units. Therefore, size of input will not matter.

• I think you should elaborate on your points a little bit more. – DuttaA Jan 28 '19 at 12:58

i think you can resize pixel your picture and convert rgb to binary or integer for the next step

• Hi and welcome to the AI Stack Exchange. Could you please expand your answer? Consider adding more context to back up your answer and perhaps adding references. – Jaden Travnik Apr 22 at 14:53

Try resizing the image to the input dimensions of your neural network architecture(keeping it fixed to something like 128*128 in a standard 2D U-net architecture) using nearest neighbor interpolation technique. This is because if you resize your image using any other interpolation, it may result in tampering with the ground truth labels. This is particularly a problem in segmentation. You won't face such a problem when it comes to classification.

Try the following:

import cv2
resized_image = cv2.resize(original_image, (new_width, new_height),
interpolation=cv2.INTER_NEAREST)