I am currently trying to improve a CNN architecture that was proposed for generating depth images. The architecture was originally proposed for autonomous driving and it looks like following :
The idea behind this architecture is to improve the accuracy of depth images by adding confidence maps to the outputs of different sensors to govern their influence on the result for each pixel. Input is an RGB image and its corresponding LIDAR data, output is a depth image with the same dimensions.
For example, RGB features are best at discriminating global information such as sky, land, body of water, etc. while LIDAR features are better at capturing fine-grained, local information. So to decide which features will have more influence at the final regression result of which pixel, scientists proposed a confidence-guided architecture where confidence weights for each map are learned during training.
Judging by their test results and how successful their paper was, their idea worked out pretty well for their problem domain. That's why I would like to employ the idea of confidence weights in my own domain, where I have multiple sources of features too. I have implemented the same architecture and got promising results: Accuracy have been improved, but not enough to compete with SOTA.
However, I believe that the architecture above can be improved for my needs. The diversity of the scene structure in my domain adds some amount of complexity to the image generation problem: Opposed to depth image generation for autonomous driving where scene structure is somewhat restricted (there is sky, there is road, there are sideways etc), the images that I need to analyze are sometimes taken from handheld cameras, some other images are areal views.
Here is a typical example of their scenario and some examples related to my domain, respectively.
This means my CNN needs to learn confidence weights for regions in fundamentally different scene setups. Top of the image can be sky, but can also be populated by people. Bottom of the image can be sea, or can be road. This brings me to the understanding that there is a non-linear relation between the position of the segments and confidence weights in my case; and I need to modify the CNN architecture by introducing some additional non-linearity to learn the confidences for different CNN columns in each situation correctly.
TLDR; I want to improve the CNN architecture above by introducing additional non-linearity, but I do not know how to do it. I have tried adding another layer to confidence weights (Extended the architecture by duplicating the same weights and activating with ReLU), but it has decreased the accuracy of the resulting model. Using the confidence weights as-is increases the accuracy, but not as much as I need.