# how to handle rectangle images in neural network?

Almost all the neural network architecture I have come across have a square input size of an image. like 32x32,64x64,128x128,.......

Ideally we might not have a square image for all kind of scenarios.

Example:384x256

My question is how to we handle such images during

1. training
2. Development
3. Test

of a neural network?

Do we force the image to resize to the input of the neural network or just crop the image to the required input size?

PS: Have asked the same on Coursera

• I haven't come across any theoretical material which suggests that images have to be square....I suggest you go through some introductory materials on CNN... It'll help you gain a better understanding – DuttaA Oct 9 '18 at 12:52
• I think it's more a choice for simplicity – Jérémy Blain Oct 9 '18 at 12:53
• @DuttaA That's exactly, why i have doubt like this! Please share your knowledge. – Santhosh Dhaipule Chandrakanth Oct 9 '18 at 13:13
• As far as I understand you think squared images are always input to a CNN which is false....Every introductory material on CNN starts with addressing the problem of high dimensions(too much input dimensions)... so it is quite difficult to answer your question without knowing the pivot of your knowledge I.e. where to begin? Since the question is the beginning of CNN architectures – DuttaA Oct 9 '18 at 14:02

## 2 Answers

I think squared image is more a choice for simplicity.

There are two types of Convolutionnal Neural Network, ones with fully connected network at the end (usually called CNN) and ones with only convolutionnal layers (which are called FCN for Fully Convolutionnal Network)

With real CNN, you always have to take the same images shape, because you flatten the last convolutionnal layer, with a fixed size. As the flatten layer has a fixed size, the feature map shape from the layer before has to be the same shape, and so, inputs (images) has to.

In FCN however, you don't flatten the last convolutionnal layer, so you don't need a fixed feature map shape, and so, don't need an input fixed size.

In both cases, you don't need squared image. You just have to be careful in the case you use CNN with full connected layer, to have the right shape for the flatten layer.

For instance, if you have an input size 320x160, and you have 3 pooling layers, so your output in the last convolutionnal layer is 40x20xc (with c the number of filters/channels) then you just need the flatten layer to have 40*20*c neurons.

If you create a new network, just design it to handle rectangle image.

If you want to use an already created, and pretrained one, I think the better choice is to resize the image. I am not sure, but if information are important and you delete them by cropping, maybe your prediction can be wrong (it depends if the object of interest is in the parts of the image that is cropped)... Actually, in Yolo (an object recognition network), images are resized if they don't fit the inputs requirement. See figure 1 of the YOLO paper It's because you don't need a high resolution to detect an object (for example the CIFAR dataset have image of 32x32, but the network can still predict correct label). So I think resize your image don't affect the prediction much (unless the new size is very different from the original)

• Yes you are right, but which gives a better result resize or crop? – Santhosh Dhaipule Chandrakanth Oct 9 '18 at 13:20
• Do you mean, if you have to (because of a CNN) to use a squared image, what is the best chice ? – Jérémy Blain Oct 9 '18 at 13:22
• It's almost, always a square. – Santhosh Dhaipule Chandrakanth Oct 9 '18 at 13:29
• @SanthoshDhaipuleChandrakanth I edited theis answer to better answer your question. If it's not clear or want more reference, ask me. – Jérémy Blain Oct 9 '18 at 13:40
• I think The user saw too many examples of use of squared images and so somehow thinks squared images is the norm...It is better the OP go through the CNN literature because it's a kind of redundant question – DuttaA Oct 9 '18 at 13:57

If you have a rectangular image and you are using existing models (or existing code) then you have to add an input pre-processing pipeline which transforms the image to standard dimensions. This is very common in computer vision and both PyTorch and Tensorflow have support for easily adding input pre-processing input pipeline for such a transformation.

Also, if you have a fixed size rectangular image data then you can design your own network architecture (or initial module) which takes image features into account by using asymmetric pooling and convolutions.