I am performing a regression task on sparse images. The images are a result of a physical process with meaningful parameters (actually, they are a superposition of cone-like shapes), and I am trying to train a regressor for these parameters.
Here, sparse images mean that the data, and thus the expected output, is made of 2D square tensors, with only one channel, and that it is expectd that roughly 90% of the image is equal to zero. However, in my system, the data is represented as dense tensors.
I built a neural network with an encoder mapping the image on an output for which I have chosen activation and shape such that it corresponds with those meaningful parameters.
I then use custom layers to build an image from these parameters in a way that matches closely the physical process, and train the network by using the L2 distance between the input image and the output image.
However, for a large set of parameters, the output image will be equal to zero, since these are sparse images. This is the case in general for the initial network.
Is it possible that, through training, the neural network will learn its way out of this all-zero parameterization ?
My intuition is that, in the beginning, the loss will be equal to the L2 norm of the input image, and the gradient will be uniformly zero, hence, no learning.
Can anyone confirm ?