2
$\begingroup$

It is mentioned by Fu 2019 that overfitting might have a negative effect on training DQN. They showed that with either early stopping or experience replay this effect could be reduced. The first is reducing overfitting, the latter is increasing data.

It doesn't only have negative effects on the returns though, my test shows that it has a negative effect on value errors as well (diff. between predict V and ground truth V). I observed frequently with limited data that the training diverged almost 100% of the time (on small nets). Since increasing the amount of data could reduce the chance of divergence, I think this is an effect from overfitting.

Overfitting should mean low training loss, however, my observation is that there is a strong correlation between TD loss and value error. That is if I see a jump in TD loss, I could expect to see a jump in value error around that moment.

Or it is not overfitting because it is not really fit (i.e. high loss) but over-optimization that is for sure.

Now the question is why?

There are two points:

  • If it is overfitting, overfitting should have a positive effect because remembering values for all training states correctly is hardly a bad thing. (In fact, my training data is a superset of my testing data, so remembering should be fine.)
  • If it doesn't fit, this begs a question what over-optimization really does. It doesn't seem to fit, but it does have a negative effect. How could that be?
Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Is your training data really a superset of the test data? If so, that defeats the purpose of having the test data - you should never include your test data in your training set! Otherwise, you have no unbiased means of determining whether you're overfitting or not. $\endgroup$
    – Nuclear Hoagie
    Commented Apr 30, 2019 at 15:44
  • $\begingroup$ @NuclearWang I only use the test data to calculate value error. Anyways, overfitting or not could not be seen by looking at the value error since the optimization is done in TD loss. I expect to see low TD loss. Even in this training data, TD loss is also increasing contradicting to a general idea of overfitting. I should point out that increasing TD loss could be a side effect of divergence. $\endgroup$
    – Phizaz
    Commented May 1, 2019 at 2:54
  • $\begingroup$ Then what do you mean by superset? Simply memorizing the training set is usually a bad thing, since there's no generalization to unseen data. But you seem to be saying that it's not a problem because your training data points are part of the test set. Low value error on the training set doesn't necessarily indicate overfitting, that's only the case if you also see high value error on the test set. If value error on the train and test is similar then the model is not overfit, and that can happen with a both a low-error (good) and a high-error (poor) model. $\endgroup$
    – Nuclear Hoagie
    Commented May 1, 2019 at 12:54
  • $\begingroup$ @NuclearWang You seem to suggest that I cannot say the word "overfit" because I don't have a proper test data. You are right on that. I kinda assume that to be the case because I do train looooong enough. Saying it is overfitting due to loss is hard in RL with bootstrapping because I always see divergence before the end of training, not having a chance to see loss coverge to zero. I now hypothesize that maybe overfitting causes divergence. That's why overfitting is bad in RL. Not in a sense that generalization is bad in general. $\endgroup$
    – Phizaz
    Commented May 2, 2019 at 15:26

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .