0
$\begingroup$

I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, the loss goes down to 0.089, but when I try to predict on test-set, every samples have the same values.

I try to add tanh activation in the last layer and switch to roberta base model, this time the model predict different values, on the train set, it predicts positive and negative values but on the test-set, it only produces positive ones.

Is there any way to train a regression model with negative values that is more stable?

$\endgroup$
1
  • $\begingroup$ Sounds pathological. I Would plot the data that you are using for your targets in a NIST 4-plot. A little bit of gross reality checking can save you some grief. There is pre-processing that takes things like the mean and the standard deviation away from the learner, so it can major on the hard parts. I would also look for indexing issues in your code, things where you are thinking your for loop is changing the variable “i” but the variable is only looking at “1“ or similar. $\endgroup$ Commented Jun 4, 2022 at 21:01

1 Answer 1

1
$\begingroup$

A couple things you could try:

You could try normalizing your target variable, so that it's number of standard deviations from the mean, or mapped to [-1,1].

If you are using drop-out during training. Make sure you turn it off during inference time.

Verify something isn't funky with your test set. Combine your training and test set and re-split it. Are you seeing the same behavior?

1 epoch isn't much, what happens when you continue training?

$\endgroup$
5
  • $\begingroup$ Yes, my targets values range from -1 to 1, the average loss in epoch 2 just stayed around 0.089 with slightly drecreasing,not fast like epoch 1 so I stopped it, I also use model.eval() when predicting so I assume that dropout is not a cause. $\endgroup$ Commented Jun 4, 2022 at 16:32
  • $\begingroup$ What happens when you resplit your dataset? Same thing? Are you certain you are doing the same preprocessing on the train and test set? $\endgroup$ Commented Jun 6, 2022 at 6:47
  • $\begingroup$ same preprocessing because they are just texts, so I just clean some url links and stop words $\endgroup$ Commented Jun 9, 2022 at 2:30
  • $\begingroup$ But I really change the structure of label and input. This task actually is about order ranking. I calculate relative rank of one item compared with the others by the formulas: d(rank_i,rank_j) = rank_i -rank_j, so if I want to know the order of rank_j relative to all of item i’s in dataset, I just need to subtract and find all the d’s.That’s why my dataset labels have a lot of negative and positive values and it ranges from -1 to 1. The problem may be I’ve changed the structure and the model does not see any pattern in it? $\endgroup$ Commented Jun 9, 2022 at 2:38
  • $\begingroup$ Where are you getting ground truth for the ranks? When you say relative to "all of items i's in dataset" what do you mean exactly? What dataset ... train, test or both. $\endgroup$ Commented Jun 12, 2022 at 8:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .