I'm currently training a CNN + multiple target regression model that does the following
input: $ \dim x = (L, 2), \text{where} \ x_i \in (-0.1, 0.1) $
output: $\dim y = (M), \text{where} \ y_i \geq 0, y_{i-1} \leq y_i$
with $10< L, M < 100$
The basic architecture is (each step with ReLU activation) :
input $\rightarrow$ CNN1(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ CNN2(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ flatten $\rightarrow$ hidden1 $\rightarrow$ hidden2 $\rightarrow$ M outputs
The issue is that the model is completely insensitive to the variations in the input data, i.e., regardless of the input value, the model would predict an output that is almost the average of all training outputs.
Some hyperparameters:
batch = 20
epoch = 5
lr = 0.05
decay = 0.02
momentum = 0.9
The training errors are not really lowering after the first few epochs, so I think it could be stuck in a local minima.
Here I'm plotting 10 random predictions on the test set data
Looks to me that there is not enough distinction in the inputs for the model to recognize, or that the input simply looks like noise to the model. I've tried some rudimentary data augmentation techniques, exponentiating the differences, normalization, etc. to no avail.
The training data is generated by diagonalizing matrices parametrized by input values, and the lowest $M$ eigenvalues are the output values.
Since my background is not in the field, I would appreciate any advice and suggestions. Thanks in advance.
