There are different questions and even different lines of thought here. Let's go through them
- Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN)
- What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small image VS big image). Take into account that the kernel sizes do not vary although the image size does.
On keeping aspect ratio (or letterbox as some people like to say)
Why to keep aspect ratio? This is more of a philosophical question. It is believed that keeping the aspect ratio helps the nets to learn the natural variability in object sizes (say a person bounding box can not be super hight and super thin, because that would be a street light).
Why not to keep aspect ratio? If you resize without keeping the aspect ratio and the aspect ratio distortion is not mega super very huge, the networks will still learn. In other words, if your input images don't have crazy aspect ratios, then there is no difference between adding or not a bit of distortion. In fact, sometimes it will even act as a regularization or augmentation.
As long as your application is not too specific and your input images aspect ratios are bounded (this is, if you train with images from any regular camera), you should not worry too much about this.
When to worry about this? When you train with huge vertical or horizontal images or if you train with images taken from some very specific devices like geophisical, radio, or optical sensors. In these cases you should pay special attention on how to resize or split an image. For example, with a recording of a radio sensor, if you resize with aspect ratio deformation, a wave from a specific frequency would transform to another because of the sine wave warp)