Why do we resize images before using them for object detection?

Question

In object detection, we can resize images by keeping the ratio the same as the original image, which is often known as "letterbox" resize.

My questions are

1. Why do we need to resize images? If we resize images to have all the same dimensions, given that some original images are too long vertically or horizontally, we will lose a lot of features in those images.

2. If the "letterbox" method is better than "normal resize" (i.e. without keeping the aspect ratio, e.g. the result of the application of OpenCV's resize function with the parameter interpolation set to cv2.INTER_AREA), why don't people apply it in the classification task?

Answer 1

There are different questions and even different lines of thought here. Let's go through them

On resizing

• Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN)
• What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small image VS big image). Take into account that the kernel sizes do not vary although the image size does.

On keeping aspect ratio (or letterbox as some people like to say)

• Why to keep aspect ratio? This is more of a philosophical question. It is believed that keeping the aspect ratio helps the nets to learn the natural variability in object sizes (say a person bounding box can not be super hight and super thin, because that would be a street light).

• Why not to keep aspect ratio? If you resize without keeping the aspect ratio and the aspect ratio distortion is not mega super very huge, the networks will still learn. In other words, if your input images don't have crazy aspect ratios, then there is no difference between adding or not a bit of distortion. In fact, sometimes it will even act as a regularization or augmentation.

Conlusion

As long as your application is not too specific and your input images aspect ratios are bounded (this is, if you train with images from any regular camera), you should not worry too much about this.

When to worry about this? When you train with huge vertical or horizontal images or if you train with images taken from some very specific devices like geophisical, radio, or optical sensors. In these cases you should pay special attention on how to resize or split an image. For example, with a recording of a radio sensor, if you resize with aspect ratio deformation, a wave from a specific frequency would transform to another because of the sine wave warp)