3
$\begingroup$

So this is my current result (loss and score per episode) of my RL model in a simple two players game:

enter image description here

I use DQN with CNN as a policy and target networks. I train my model using Adam optimizer and calculate the loss using Smooth L1 Loss.

In a normal "Supervised Learning" situation, I can deduce that my model is overfitting. And I can imagine some methods to tackle this problem (e.g. Dropout layer, Regularization, Smaller Learning Rate, Early Stopping). But is that solution will also work in RL problem? Or are there any better solutions to handle overfitting in RL?

$\endgroup$
3
$\begingroup$

Overfitting refers to a model being stuck in a local minimum while trying to minimise a loss function. In Reinforcement Learning the aim is to learn an optimal policy by maximising (not always but almost) a reward function rather than minimising a loss function, so you can't really talk of overfitting as in the supervised scenario, but you can definitely talk about sub-optimal policies.

If we think of a specific task like avoiding stationary objects, a simple sub-optimal policy would be to just stay still without moving at all, or moving in circles if the reward function was designed to penalise lack of movements.

The way to avoid an agent to learn sub-optimal policies is to find a good compromise between exploitation, i.e. the constant selection of the next action to take based on the maximum expected reward possible, and exploration, i.e. a random selection of the next action to take regardless of the rewards. Here's a link to an introduction to the topic: Exploration and Exploitation in Reinforcement Learning

It is worth mentioning that sometimes an agent can actually outsmart humans though, some examples are reported in this paper The Surprising Creativity of Digital Evolution. I particularly like the story of the insect agent trained to learn to walk while minimising the contact with the floor surface. The agent surprisingly managed to learn to walk without touching the ground at all. When the authors checked what was going on they discovered that the insect leaned to flip itself and then walk using its fake 'elbows' (fig7 in the linked paper). I add this story just to point out that most of the time the design of the reward function is itself even more important than exploration and exploitation tuning.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I wouldn't describe overfitting as the situation where a model gets stuck in a local minimum, but as the "gap between training and test performances". This definition should also make it clearer why it may not make sense to talk about overfitting in RL. If we use your description of overfitting, one could argue that "sub-optimal policies" correspond to local minima, and they wouldn't be wrong. I think it would also be useful you if commented the plots that the asker provided. What do thee plots represent and can we understand whether the RL agent is "overfitting" from them? $\endgroup$ – nbro Apr 9 at 16:33
  • $\begingroup$ ah... I see.. so can you help me to understand why my Agent's Score decreasing after 400 episodes? it keeps decreasing until almost 0 after 1000 episodes. Because in this game the reward is the score (0 to 12), so I think the problem is not a sub-optimal policy, right? $\endgroup$ – malioboro Apr 10 at 6:02
  • $\begingroup$ @malioboro Could you explain a bit more in details the game and what reward function you're using? Honestly the only thing I can tell from your graphs is that after 200 iterations your neural net starts overfitting, affecting the performances of the agent in selecting the next action. You could just stop training the deep net when the loss start increase, and keep using the same model to train the policy of your agent. Also, consider that sometimes it can take more than 1k, even 10k iterations to reach an optimal policy, the drop you see might be just a random fluctuation. $\endgroup$ – Edoardo Guerriero Apr 10 at 13:23
  • $\begingroup$ sorry about the last sentence, I read the comment quickly, if after 1k iterations the reward approach zero then I think the problem is most definitely related to the performance of the neural net. $\endgroup$ – Edoardo Guerriero Apr 10 at 13:40
  • $\begingroup$ @EdoardoGuerriero thank you so much for your input, I solve this problem by tune the RL model to use more steps before update the target network, and also tune the NN parameters as you said $\endgroup$ – malioboro Apr 16 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.