I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, the loss goes down to 0.089, but when I try to predict on test-set, every samples have the same values.
I try to add tanh activation in the last layer and switch to roberta base model, this time the model predict different values, on the train set, it predicts positive and negative values but on the test-set, it only produces positive ones.
Is there any way to train a regression model with negative values that is more stable?