I have a system that takes 32 inputs (all of which are 1 or 0) and generates 32 outputs (all of which are complex numbers that lie roughly in the range of (0,2)). The response of this system to its inputs are linear (meaning that its response is equivalent to multiplying with a 32*32 complex matrix).
This system is well-defined by 256 "settings", all of which real numbers in the range (0,1). But its properties are non-linear with the settings, meaning that the response matrix mentioned would not change linearly according to these settings.
My goal is to use some kind of neural network to predict this response matrix using the settings as the only input.
The dataset consists of 50000 (can be extended if necessary) input-output-settings tuples, which of course doesn't allow me to compute the target response matrix exactly. Moreover, input patterns come in a sparse form, meaning that, for most of the time, input vectors are mostly zeroes with a small number of ones.
My approach was to use ComplexCNN with sigmoid to generate the response matrix, then multiplying it with the input vectors to get a predicted output, comparing it with those from the dataset, and computing a loss (using basic $L_1$ loss). I implemented and trained the model using PyTorch, but results came out very badly, as the input, output, and matrix are sparse, causing gradient problems, which often lead to the neural network to give out
nan as output.
So, might there be any way to change the network architecture to adapt to this type of data? How can I solve the problem of non-converging gradients?