Best practice for handling letterboxed images for non fully-convolutional deep learning networks?

Question

I'm working on a depth estimation network. It has two outputs:

1. A relative depth map
2. A scalar for scaling the relative depth map into an absolute depth map. This second output uses dense layers so we cannot use variable-sized input.

We are trying to handle two different dimensions (192x256 and 256x192). The current approach is to letterbox the image, meaning apply black on the image so that it comes out to 256x256. We decided on this approach instead of center-cropping images to 192x192 because we believe we may lose valuable data with cropping.

When using letterboxes, I see two paths:

1. Ignore the letterbox portions of the image in my loss function. The loss function will only perform calculations on the original portion of the image.
2. Set a static value for the letterbox portion and include it as part of the loss.

Is #1 the correct approach? The network will then be able to predict any depth value for the black letterbox portions without being penalized. I'm concerned with #2 about confusing the network between the letterbox portion and actual dark portions of images.

Answer 1

Padding is indeed the easiest solution. And if no bias is used then masking the extra values during the loss computation is also not necessary, since it's enough to use zero as padding value.

You might be interested though in checking Spatial Pyramid Pooling. This pooling method allows to combine fully convolutional modules and dense layers, i.e, it can be initialized to produce a specific fixed output size while allowing varying input sizes, for both, training and inference.