I'm working on a depth estimation network. It has two outputs:
- A relative depth map
- A scalar for scaling the relative depth map into an absolute depth map. This second output uses dense layers so we cannot use variable-sized input.
We are trying to handle two different dimensions (192x256 and 256x192). The current approach is to letterbox the image, meaning apply black on the image so that it comes out to 256x256. We decided on this approach instead of center-cropping images to 192x192 because we believe we may lose valuable data with cropping.
When using letterboxes, I see two paths:
- Ignore the letterbox portions of the image in my loss function. The loss function will only perform calculations on the original portion of the image.
- Set a static value for the letterbox portion and include it as part of the loss.
Is #1 the correct approach? The network will then be able to predict any depth value for the black letterbox portions without being penalized. I'm concerned with #2 about confusing the network between the letterbox portion and actual dark portions of images.