1
$\begingroup$

I'm currently training a CNN + multiple target regression model that does the following

input: $ \dim x = (L, 2), \text{where} \ x_i \in (-0.1, 0.1) $
output: $\dim y = (M), \text{where} \ y_i \geq 0, y_{i-1} \leq y_i$
with $10< L, M < 100$

The basic architecture is (each step with ReLU activation) :
input $\rightarrow$ CNN1(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ CNN2(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ flatten $\rightarrow$ hidden1 $\rightarrow$ hidden2 $\rightarrow$ M outputs

The issue is that the model is completely insensitive to the variations in the input data, i.e., regardless of the input value, the model would predict an output that is almost the average of all training outputs.

Some hyperparameters:

batch = 20
epoch = 5
lr = 0.05
decay = 0.02
momentum = 0.9

The training errors are not really lowering after the first few epochs, so I think it could be stuck in a local minima.

Here I'm plotting 10 random predictions on the test set data Plotting 10 random predicts on the test set data

Looks to me that there is not enough distinction in the inputs for the model to recognize, or that the input simply looks like noise to the model. I've tried some rudimentary data augmentation techniques, exponentiating the differences, normalization, etc. to no avail.

The training data is generated by diagonalizing matrices parametrized by input values, and the lowest $M$ eigenvalues are the output values.

Since my background is not in the field, I would appreciate any advice and suggestions. Thanks in advance.

$\endgroup$
3
  • $\begingroup$ for how many epochs are you training it? with which optimizer/learning rate? not clear what you are doing, but are you sure that there is enough information in the input to predict the output? $\endgroup$
    – Alberto
    Aug 6, 2023 at 10:30
  • $\begingroup$ @AlbertoSinigaglia Please see the parameters above. As the answer below suggested, a larger learning rate sometimes results in weights becoming NaN, presumably some gradient explosion (?) The inputs generates a Hermitian matrix (they alone do not generate the matrix, but the additional information needed for such generation is identical across all cases), and I'm simulating the process of solving (diagonalizing) that matrix, but that's a good question. I will look into that more closely $\endgroup$
    – RLLL
    Aug 8, 2023 at 4:06
  • $\begingroup$ What loss function did you use? This could be the key here $\endgroup$
    – Ggjj11
    May 4 at 21:10

1 Answer 1

0
$\begingroup$

It seems like the model is falling into a bad local minimum, where the model is stuck in an easy-to-learn "good-enough" solution. Increasing the initial learning rate could fix this.

See this post for more information on this.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .